2024-04-25T15:13:09.168 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T15:13:09.169 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201 -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/orig.config.yaml 2024-04-25T15:13:09.212 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:13:09.295 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} email: pdonnell@redhat.com first_in_suite: false job_id: '7673201' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQcOlVxd1/CY5+GNNxG/GW2bWEGfxv1T+F972WipdFff3Uoq5q2kf5snnF+Wr0YvTlLWQb0yOwELfzjQyBMzQ8= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3cCg9gTCONg9DJ8ATtl2/Liuyg4DoCd5g0p6RrKYZZBcBCL5Xf3J/DMVLliAWXV3y76EzJas7ENCk58VwXd18= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_quiesce teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T15:13:09.295 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T15:13:09.296 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T15:13:09.296 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T15:13:09.297 INFO:teuthology.task.internal:Checking packages... 2024-04-25T15:13:09.318 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T15:13:09.318 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T15:13:09.318 INFO:teuthology.packaging:ref: None 2024-04-25T15:13:09.318 INFO:teuthology.packaging:tag: None 2024-04-25T15:13:09.318 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:13:09.318 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:13:09.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:13:09.583 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T15:13:09.585 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T15:13:09.591 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T15:13:09.591 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T15:13:09.610 INFO:teuthology.task.internal:Saving configuration 2024-04-25T15:13:09.625 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T15:13:09.631 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T15:13:09.653 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:07:57.871572', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQcOlVxd1/CY5+GNNxG/GW2bWEGfxv1T+F972WipdFff3Uoq5q2kf5snnF+Wr0YvTlLWQb0yOwELfzjQyBMzQ8='} 2024-04-25T15:13:09.674 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:07:57.873359', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3cCg9gTCONg9DJ8ATtl2/Liuyg4DoCd5g0p6RrKYZZBcBCL5Xf3J/DMVLliAWXV3y76EzJas7ENCk58VwXd18='} 2024-04-25T15:13:09.674 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T15:13:09.681 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-25T15:13:09.681 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-25T15:13:09.681 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T15:13:09.754 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8af1495f70>, signals=[15]) 2024-04-25T15:13:09.755 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T15:13:09.761 INFO:teuthology.task.internal:Opening connections... 2024-04-25T15:13:09.761 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:13:09.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:13:09.837 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:13:09.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:13:09.910 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T15:13:09.917 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-04-25T15:13:09.935 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-04-25T15:13:09.935 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-04-25T15:13:09.992 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-04-25T15:13:09.992 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:13:09.993 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:13:09.994 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-04-25T15:13:10.018 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-25T15:13:10.036 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-25T15:13:10.036 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-25T15:13:10.093 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-04-25T15:13:10.093 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-04-25T15:13:10.093 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-04-25T15:13:10.093 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:13:10.093 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:13:10.094 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:13:10.094 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-25T15:13:10.118 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T15:13:10.129 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T15:13:10.135 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T15:13:10.135 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:13:10.137 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:13:10.153 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T15:13:10.159 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T15:13:10.159 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:13:10.195 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:13:10.283 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T15:13:10.342 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T15:13:10.355 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T15:13:10.356 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T15:13:10.356 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:13:10.356 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:13:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:13:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:13:10.374 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-04-25T15:13:10.374 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-04-25T15:13:10.430 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-437.el9.x86_64 2024-04-25T15:13:10.430 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-437.el9.x86_64 2024-04-25T15:13:10.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-04-25T15:13:10.432 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-437.el9.x86_64 2024-04-25T15:13:10.432 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-437.el9.x86_64 2024-04-25T15:13:10.432 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-04-25T15:13:12.791 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.0 MB 00:00 2024-04-25T15:13:12.881 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.0 MB 00:00 2024-04-25T15:13:14.559 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-25T15:13:14.774 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-25T15:13:18.609 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-25T15:13:18.966 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-25T15:13:20.383 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-25T15:13:20.655 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-25T15:13:21.543 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-25T15:13:21.707 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-25T15:13:26.625 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-25T15:13:26.809 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 9.9 kB/s | 1.7 kB 00:00 2024-04-25T15:13:28.500 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T15:13:28.501 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T15:13:28.583 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-25T15:13:28.584 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-04-25T15:13:28.584 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-25T15:13:28.693 DEBUG:teuthology.orchestra.run.smithi177:> echo no | sudo yum reinstall kernel || true 2024-04-25T15:13:28.698 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T15:13:28.698 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T15:13:28.791 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-25T15:13:28.792 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-04-25T15:13:28.792 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-04-25T15:13:28.888 DEBUG:teuthology.orchestra.run.smithi146:> echo no | sudo yum reinstall kernel || true 2024-04-25T15:13:29.190 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:13:26 PM UTC. 2024-04-25T15:13:29.314 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-25T15:13:29.326 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-25T15:13:29.326 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:Reinstalling: 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 255 k 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 0 2024-04-25T15:13:29.327 INFO:teuthology.orchestra.run.smithi177.stderr:Operation aborted. 2024-04-25T15:13:29.380 INFO:teuthology.orchestra.run.smithi177.stdout:Is this ok [y/N]: 2024-04-25T15:13:29.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum reinstall -y kernel || true 2024-04-25T15:13:29.395 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:13:26 PM UTC. 2024-04-25T15:13:29.522 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:Reinstalling: 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 255 k 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 0 2024-04-25T15:13:29.531 INFO:teuthology.orchestra.run.smithi146.stderr:Operation aborted. 2024-04-25T15:13:29.587 INFO:teuthology.orchestra.run.smithi146.stdout:Is this ok [y/N]: 2024-04-25T15:13:29.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum reinstall -y kernel || true 2024-04-25T15:13:29.874 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:13:26 PM UTC. 2024-04-25T15:13:29.991 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-25T15:13:29.993 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-25T15:13:29.993 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-04-25T15:13:29.993 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-25T15:13:29.993 INFO:teuthology.orchestra.run.smithi177.stdout:Reinstalling: 2024-04-25T15:13:29.993 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 255 k 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 0 2024-04-25T15:13:29.994 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-25T15:13:30.082 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 03:13:26 PM UTC. 2024-04-25T15:13:30.196 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-25T15:13:30.197 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-25T15:13:30.197 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-04-25T15:13:30.197 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout:Reinstalling: 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 255 k 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 0 2024-04-25T15:13:30.198 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-04-25T15:13:30.254 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-25T15:13:30.255 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:13:30.255 INFO:teuthology.orchestra.run.smithi177.stdout:Total 971 kB/s | 255 kB 00:00 2024-04-25T15:13:30.273 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-25T15:13:30.466 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-25T15:13:30.466 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-25T15:13:30.471 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-25T15:13:30.471 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-25T15:13:30.607 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-25T15:13:30.610 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-04-25T15:13:30.611 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:13:30.611 INFO:teuthology.orchestra.run.smithi146.stdout:Total 616 kB/s | 255 kB 00:00 2024-04-25T15:13:30.631 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-04-25T15:13:30.688 INFO:teuthology.orchestra.run.smithi177.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:13:30.689 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:13:30.803 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-04-25T15:13:30.803 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-04-25T15:13:30.808 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-04-25T15:13:30.808 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-04-25T15:13:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-04-25T15:13:31.012 INFO:teuthology.orchestra.run.smithi146.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:13:31.012 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:13:31.341 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:13:31.341 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:13:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:13:31.733 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:13:32.045 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:13:32.046 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:13:32.046 INFO:teuthology.orchestra.run.smithi177.stdout:Reinstalled: 2024-04-25T15:13:32.046 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T15:13:32.046 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:13:32.046 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-25T15:13:32.208 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T15:13:32.255 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T15:13:32.255 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T15:13:32.255 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T15:13:32.255 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T15:13:32.255 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T15:13:32.292 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-04-25T15:13:32.310 DEBUG:teuthology.parallel:result is None 2024-04-25T15:13:32.424 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:13:32.424 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:13:32.424 INFO:teuthology.orchestra.run.smithi146.stdout:Reinstalled: 2024-04-25T15:13:32.424 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T15:13:32.424 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:13:32.424 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-04-25T15:13:32.590 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T15:13:32.636 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T15:13:32.637 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T15:13:32.637 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T15:13:32.637 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T15:13:32.637 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T15:13:32.671 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-04-25T15:13:32.689 DEBUG:teuthology.parallel:result is None 2024-04-25T15:13:32.689 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T15:13:32.696 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T15:13:32.697 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:13:32.732 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:13:32.747 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T15:13:32.752 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T15:13:32.780 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T15:13:32.781 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:13:32.787 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:13:32.815 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T15:13:32.821 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T15:13:32.821 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:13:32.849 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:13:32.875 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:13:32.883 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:13:32.884 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:13:32.893 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:13:32.894 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T15:13:32.902 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T15:13:32.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:13:32.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:13:32.959 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T15:13:32.968 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T15:13:32.968 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:13:32.992 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:13:33.014 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:13:33.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:13:33.131 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:13:33.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:13:33.252 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:13:33.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:13:33.315 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:13:33.355 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:13:33.400 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:13:33.484 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:13:33.520 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:13:33.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:13:33.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-04-25T15:13:33.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-25T15:13:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:13:33.664 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:13:34.000 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T15:13:34.009 INFO:teuthology.task.internal:Starting timer... 2024-04-25T15:13:34.009 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T15:13:34.038 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T15:13:34.047 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-25T15:13:34.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-04-25T15:13:34.095 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-04-25T15:13:34.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-04-25T15:13:34.135 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-04-25T15:13:34.136 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T15:13:34.137 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-04-25T15:13:34.163 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-04-25T15:13:34.163 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-04-25T15:13:34.200 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-04-25T15:13:34.200 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi146.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-04-25T15:13:34.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:13:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:13:34.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:13:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:13:34.264 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T15:13:34.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:13:34.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:13:34.328 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T15:13:34.338 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T15:13:34.355 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T15:13:34.356 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi146.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T15:17:03.129 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-04-25T15:17:03.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-25T15:17:03.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:17:03.207 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-04-25T15:17:03.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-25T15:17:03.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-25T15:17:03.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:17:03.364 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-25T15:17:03.442 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-25T15:17:03.443 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T15:17:03.453 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T15:17:03.453 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:17:03.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:17:03.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:17:03.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:17:03.481 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:17:03.495 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:17:03.512 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:17:03.517 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-04-25T15:17:03.526 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:17:03.527 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-04-25T15:17:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:17:03.549 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-04-25T15:17:03.553 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:17:03.560 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-04-25T15:17:03.573 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:17:03.586 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:17:03.614 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:17:03.618 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:17:03.618 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-04-25T15:17:03.618 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.618 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.618 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.618 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.639 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:17:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:17:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-25T15:17:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.643 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:17:03.644 INFO:teuthology.run_tasks:Running task install... 2024-04-25T15:17:03.652 DEBUG:teuthology.task.install:project ceph 2024-04-25T15:17:03.652 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T15:17:03.652 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T15:17:03.652 INFO:teuthology.task.install:Using flavor: default 2024-04-25T15:17:03.667 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T15:17:03.667 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T15:17:03.667 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:17:03.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:17:03.670 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:17:03.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:17:04.008 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:17:04.008 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:17:04.026 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:17:04.027 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:17:04.269 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:17:04.269 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:17:04.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:17:04.289 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:17:04.289 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:17:04.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:17:04.294 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:17:04.295 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:17:04.315 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:17:04.315 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:17:04.361 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:17:04.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:17:04.435 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:17:04.456 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:17:04.498 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-04-25T15:17:04.500 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-04-25T15:17:04.520 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-04-25T15:17:04.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-04-25T15:17:04.723 INFO:teuthology.orchestra.run.smithi177.stdout:48 files removed 2024-04-25T15:17:04.746 INFO:teuthology.orchestra.run.smithi146.stdout:48 files removed 2024-04-25T15:17:04.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:17:04.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:17:05.459 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 186 kB/s | 84 kB 00:00 2024-04-25T15:17:05.610 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 145 kB/s | 84 kB 00:00 2024-04-25T15:17:06.018 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 23 kB/s | 12 kB 00:00 2024-04-25T15:17:06.135 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-25T15:17:06.460 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-25T15:17:06.486 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-25T15:17:07.029 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-25T15:17:07.222 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T15:17:09.182 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-25T15:17:09.423 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-25T15:17:13.382 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-25T15:17:13.549 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T15:17:14.839 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-25T15:17:15.072 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-25T15:17:15.965 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-04-25T15:17:16.032 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-25T15:17:21.015 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T15:17:21.057 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T15:17:23.241 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:17:23.242 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:17:23.242 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:17:23.242 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:17:23.252 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:17:23.253 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:17:23.253 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:17:23.254 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:17:23.254 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:17:23.255 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:17:23.255 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:17:23.256 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:17:23.256 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:17:23.257 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:17:23.259 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:17:23.260 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:17:23.260 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:17:23.260 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:17:23.323 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-25T15:17:23.323 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-25T15:17:23.329 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:17:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:17:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:17:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:17:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:17:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:17:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:17:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:17:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:17:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:17:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:17:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout:Install 148 Packages 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 4 Packages 2024-04-25T15:17:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:17:23.342 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 214 M 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================= 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================= 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:17:23.343 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:17:23.344 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:17:23.345 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:17:23.346 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:17:23.347 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:17:23.348 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:17:23.349 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:17:23.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:17:23.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:17:23.352 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:17:23.353 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================= 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:Install 148 Packages 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 4 Packages 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 214 M 2024-04-25T15:17:23.354 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-04-25T15:17:24.608 INFO:teuthology.orchestra.run.smithi146.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-25T15:17:24.633 INFO:teuthology.orchestra.run.smithi177.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-25T15:17:25.034 INFO:teuthology.orchestra.run.smithi146.stdout:(2/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 6.5 MB/s | 5.4 MB 00:00 2024-04-25T15:17:25.134 INFO:teuthology.orchestra.run.smithi146.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 1.4 MB/s | 147 kB 00:00 2024-04-25T15:17:25.575 INFO:teuthology.orchestra.run.smithi146.stdout:(4/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 892 kB/s | 863 kB 00:00 2024-04-25T15:17:25.691 INFO:teuthology.orchestra.run.smithi177.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 815 kB/s | 863 kB 00:01 2024-04-25T15:17:25.734 INFO:teuthology.orchestra.run.smithi146.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 3.8 MB/s | 2.3 MB 00:00 2024-04-25T15:17:25.983 INFO:teuthology.orchestra.run.smithi177.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 504 kB/s | 147 kB 00:00 2024-04-25T15:17:26.534 INFO:teuthology.orchestra.run.smithi177.stdout:(4/152): ceph-common-19.0.0-3291.g994e2f20.el9. 8.3 MB/s | 20 MB 00:02 2024-04-25T15:17:26.618 INFO:teuthology.orchestra.run.smithi146.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 1.4 MB/s | 1.5 MB 00:01 2024-04-25T15:17:26.801 INFO:teuthology.orchestra.run.smithi177.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 5.6 MB/s | 1.5 MB 00:00 2024-04-25T15:17:27.534 INFO:teuthology.orchestra.run.smithi177.stdout:(6/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 6.7 MB/s | 4.9 MB 00:00 2024-04-25T15:17:27.993 INFO:teuthology.orchestra.run.smithi146.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 2.2 MB/s | 4.9 MB 00:02 2024-04-25T15:17:28.284 INFO:teuthology.orchestra.run.smithi177.stdout:(7/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 5.4 MB 00:04 2024-04-25T15:17:28.859 INFO:teuthology.orchestra.run.smithi177.stdout:(8/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 804 kB/s | 2.3 MB 00:02 2024-04-25T15:17:28.993 INFO:teuthology.orchestra.run.smithi177.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 189 kB/s | 25 kB 00:00 2024-04-25T15:17:29.752 INFO:teuthology.orchestra.run.smithi177.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 8.0 MB/s | 18 MB 00:02 2024-04-25T15:17:29.835 INFO:teuthology.orchestra.run.smithi177.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.6 MB/s | 224 kB 00:00 2024-04-25T15:17:29.894 INFO:teuthology.orchestra.run.smithi177.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 464 kB/s | 27 kB 00:00 2024-04-25T15:17:29.969 INFO:teuthology.orchestra.run.smithi177.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 423 kB/s | 32 kB 00:00 2024-04-25T15:17:30.103 INFO:teuthology.orchestra.run.smithi177.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 5.5 MB/s | 756 kB 00:00 2024-04-25T15:17:30.486 INFO:teuthology.orchestra.run.smithi177.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 412 kB/s | 158 kB 00:00 2024-04-25T15:17:30.553 INFO:teuthology.orchestra.run.smithi177.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.9 MB/s | 127 kB 00:00 2024-04-25T15:17:30.628 INFO:teuthology.orchestra.run.smithi177.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 6.5 MB/s | 497 kB 00:00 2024-04-25T15:17:30.768 INFO:teuthology.orchestra.run.smithi146.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 3.6 MB/s | 10 MB 00:02 2024-04-25T15:17:30.868 INFO:teuthology.orchestra.run.smithi146.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 252 kB/s | 25 kB 00:00 2024-04-25T15:17:31.020 INFO:teuthology.orchestra.run.smithi177.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 13 MB/s | 5.2 MB 00:00 2024-04-25T15:17:31.070 INFO:teuthology.orchestra.run.smithi177.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 908 kB/s | 45 kB 00:00 2024-04-25T15:17:31.146 INFO:teuthology.orchestra.run.smithi177.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.8 MB/s | 135 kB 00:00 2024-04-25T15:17:31.221 INFO:teuthology.orchestra.run.smithi177.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.1 MB/s | 161 kB 00:00 2024-04-25T15:17:31.288 INFO:teuthology.orchestra.run.smithi177.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 4.7 MB/s | 321 kB 00:00 2024-04-25T15:17:31.372 INFO:teuthology.orchestra.run.smithi177.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.5 MB/s | 298 kB 00:00 2024-04-25T15:17:31.438 INFO:teuthology.orchestra.run.smithi177.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.5 MB/s | 100 kB 00:00 2024-04-25T15:17:31.505 INFO:teuthology.orchestra.run.smithi177.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T15:17:31.922 INFO:teuthology.orchestra.run.smithi177.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 7.3 MB/s | 3.0 MB 00:00 2024-04-25T15:17:31.981 INFO:teuthology.orchestra.run.smithi177.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.9 MB/s | 169 kB 00:00 2024-04-25T15:17:32.048 INFO:teuthology.orchestra.run.smithi177.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 515 kB/s | 34 kB 00:00 2024-04-25T15:17:32.106 INFO:teuthology.orchestra.run.smithi177.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.4 MB/s | 142 kB 00:00 2024-04-25T15:17:32.474 INFO:teuthology.orchestra.run.smithi177.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 9.7 MB/s | 3.5 MB 00:00 2024-04-25T15:17:32.568 INFO:teuthology.orchestra.run.smithi146.stdout:(10/152): ceph-common-19.0.0-3291.g994e2f20.el9 2.3 MB/s | 20 MB 00:08 2024-04-25T15:17:32.574 INFO:teuthology.orchestra.run.smithi177.stdout:(31/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 2.4 MB/s | 10 MB 00:04 2024-04-25T15:17:32.710 INFO:teuthology.orchestra.run.smithi146.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.6 MB/s | 224 kB 00:00 2024-04-25T15:17:32.749 INFO:teuthology.orchestra.run.smithi177.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 1.4 MB/s | 249 kB 00:00 2024-04-25T15:17:32.794 INFO:teuthology.orchestra.run.smithi146.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 325 kB/s | 27 kB 00:00 2024-04-25T15:17:32.834 INFO:teuthology.orchestra.run.smithi177.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 590 kB/s | 49 kB 00:00 2024-04-25T15:17:32.877 INFO:teuthology.orchestra.run.smithi146.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 381 kB/s | 32 kB 00:00 2024-04-25T15:17:32.900 INFO:teuthology.orchestra.run.smithi177.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 246 kB/s | 16 kB 00:00 2024-04-25T15:17:33.119 INFO:teuthology.orchestra.run.smithi146.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 3.1 MB/s | 756 kB 00:00 2024-04-25T15:17:33.134 INFO:teuthology.orchestra.run.smithi177.stdout:(35/152): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-04-25T15:17:33.159 INFO:teuthology.orchestra.run.smithi177.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.0 MB/s | 273 kB 00:00 2024-04-25T15:17:33.186 INFO:teuthology.orchestra.run.smithi146.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.3 MB/s | 158 kB 00:00 2024-04-25T15:17:33.218 INFO:teuthology.orchestra.run.smithi177.stdout:(37/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-25T15:17:33.243 INFO:teuthology.orchestra.run.smithi177.stdout:(38/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 218 kB/s | 18 kB 00:00 2024-04-25T15:17:33.360 INFO:teuthology.orchestra.run.smithi177.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 80 kB 00:00 2024-04-25T15:17:33.361 INFO:teuthology.orchestra.run.smithi146.stdout:(16/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 2.6 MB/s | 18 MB 00:06 2024-04-25T15:17:33.387 INFO:teuthology.orchestra.run.smithi146.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 634 kB/s | 127 kB 00:00 2024-04-25T15:17:33.419 INFO:teuthology.orchestra.run.smithi177.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-25T15:17:33.444 INFO:teuthology.orchestra.run.smithi177.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-25T15:17:33.469 INFO:teuthology.orchestra.run.smithi177.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-25T15:17:33.504 INFO:teuthology.orchestra.run.smithi146.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 3.4 MB/s | 497 kB 00:00 2024-04-25T15:17:33.511 INFO:teuthology.orchestra.run.smithi177.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-25T15:17:33.537 INFO:teuthology.orchestra.run.smithi177.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T15:17:33.562 INFO:teuthology.orchestra.run.smithi177.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T15:17:33.570 INFO:teuthology.orchestra.run.smithi146.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 678 kB/s | 45 kB 00:00 2024-04-25T15:17:33.587 INFO:teuthology.orchestra.run.smithi177.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T15:17:33.612 INFO:teuthology.orchestra.run.smithi177.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T15:17:33.638 INFO:teuthology.orchestra.run.smithi177.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 13 MB/s | 689 kB 00:00 2024-04-25T15:17:33.671 INFO:teuthology.orchestra.run.smithi146.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.3 MB/s | 135 kB 00:00 2024-04-25T15:17:33.681 INFO:teuthology.orchestra.run.smithi177.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-25T15:17:33.705 INFO:teuthology.orchestra.run.smithi177.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T15:17:33.730 INFO:teuthology.orchestra.run.smithi177.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T15:17:33.746 INFO:teuthology.orchestra.run.smithi146.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.1 MB/s | 161 kB 00:00 2024-04-25T15:17:33.755 INFO:teuthology.orchestra.run.smithi177.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T15:17:33.781 INFO:teuthology.orchestra.run.smithi177.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 660 kB/s | 33 kB 00:00 2024-04-25T15:17:33.848 INFO:teuthology.orchestra.run.smithi177.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-04-25T15:17:33.871 INFO:teuthology.orchestra.run.smithi146.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 2.5 MB/s | 321 kB 00:00 2024-04-25T15:17:33.873 INFO:teuthology.orchestra.run.smithi177.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 205 kB/s | 18 kB 00:00 2024-04-25T15:17:33.890 INFO:teuthology.orchestra.run.smithi177.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 266 kB/s | 11 kB 00:00 2024-04-25T15:17:33.915 INFO:teuthology.orchestra.run.smithi177.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-04-25T15:17:33.940 INFO:teuthology.orchestra.run.smithi177.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 909 kB/s | 45 kB 00:00 2024-04-25T15:17:33.966 INFO:teuthology.orchestra.run.smithi177.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T15:17:33.980 INFO:teuthology.orchestra.run.smithi146.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 2.7 MB/s | 298 kB 00:00 2024-04-25T15:17:33.991 INFO:teuthology.orchestra.run.smithi177.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 582 kB 00:00 2024-04-25T15:17:34.008 INFO:teuthology.orchestra.run.smithi177.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 622 kB/s | 25 kB 00:00 2024-04-25T15:17:34.025 INFO:teuthology.orchestra.run.smithi177.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T15:17:34.051 INFO:teuthology.orchestra.run.smithi177.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 292 kB 00:00 2024-04-25T15:17:34.055 INFO:teuthology.orchestra.run.smithi146.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 100 kB 00:00 2024-04-25T15:17:34.076 INFO:teuthology.orchestra.run.smithi177.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T15:17:34.093 INFO:teuthology.orchestra.run.smithi177.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 35 kB 00:00 2024-04-25T15:17:34.139 INFO:teuthology.orchestra.run.smithi146.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 86 kB 00:00 2024-04-25T15:17:34.202 INFO:teuthology.orchestra.run.smithi177.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-04-25T15:17:34.278 INFO:teuthology.orchestra.run.smithi177.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-04-25T15:17:34.310 INFO:teuthology.orchestra.run.smithi177.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-25T15:17:34.344 INFO:teuthology.orchestra.run.smithi177.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 14 MB/s | 911 kB 00:00 2024-04-25T15:17:34.369 INFO:teuthology.orchestra.run.smithi177.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T15:17:34.437 INFO:teuthology.orchestra.run.smithi177.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 16 MB/s | 1.0 MB 00:00 2024-04-25T15:17:34.439 INFO:teuthology.orchestra.run.smithi146.stdout:(26/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 4.9 MB/s | 5.2 MB 00:01 2024-04-25T15:17:34.506 INFO:teuthology.orchestra.run.smithi146.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.5 MB/s | 169 kB 00:00 2024-04-25T15:17:34.512 INFO:teuthology.orchestra.run.smithi177.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 35 MB/s | 5.8 MB 00:00 2024-04-25T15:17:34.537 INFO:teuthology.orchestra.run.smithi177.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 894 kB/s | 87 kB 00:00 2024-04-25T15:17:34.573 INFO:teuthology.orchestra.run.smithi146.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 514 kB/s | 34 kB 00:00 2024-04-25T15:17:34.578 INFO:teuthology.orchestra.run.smithi177.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 6.0 MB/s | 244 kB 00:00 2024-04-25T15:17:34.629 INFO:teuthology.orchestra.run.smithi177.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 5.1 MB/s | 248 kB 00:00 2024-04-25T15:17:34.640 INFO:teuthology.orchestra.run.smithi146.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.1 MB/s | 142 kB 00:00 2024-04-25T15:17:34.697 INFO:teuthology.orchestra.run.smithi177.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 6.0 MB 00:00 2024-04-25T15:17:34.721 INFO:teuthology.orchestra.run.smithi177.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 279 kB/s | 26 kB 00:00 2024-04-25T15:17:34.747 INFO:teuthology.orchestra.run.smithi177.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-25T15:17:34.764 INFO:teuthology.orchestra.run.smithi177.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-25T15:17:34.789 INFO:teuthology.orchestra.run.smithi177.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 834 kB/s | 35 kB 00:00 2024-04-25T15:17:34.857 INFO:teuthology.orchestra.run.smithi146.stdout:(30/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 4.2 MB/s | 3.0 MB 00:00 2024-04-25T15:17:34.898 INFO:teuthology.orchestra.run.smithi177.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-04-25T15:17:34.923 INFO:teuthology.orchestra.run.smithi177.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 250 kB 00:00 2024-04-25T15:17:34.948 INFO:teuthology.orchestra.run.smithi177.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-25T15:17:34.973 INFO:teuthology.orchestra.run.smithi177.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T15:17:34.999 INFO:teuthology.orchestra.run.smithi177.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-04-25T15:17:35.024 INFO:teuthology.orchestra.run.smithi177.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T15:17:35.050 INFO:teuthology.orchestra.run.smithi177.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T15:17:35.075 INFO:teuthology.orchestra.run.smithi177.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T15:17:35.093 INFO:teuthology.orchestra.run.smithi177.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-25T15:17:35.109 INFO:teuthology.orchestra.run.smithi177.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-25T15:17:35.151 INFO:teuthology.orchestra.run.smithi177.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-25T15:17:35.209 INFO:teuthology.orchestra.run.smithi177.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 785 kB/s | 44 kB 00:00 2024-04-25T15:17:35.293 INFO:teuthology.orchestra.run.smithi177.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.7 MB/s | 221 kB 00:00 2024-04-25T15:17:35.349 INFO:teuthology.orchestra.run.smithi146.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 5.0 MB/s | 3.5 MB 00:00 2024-04-25T15:17:35.449 INFO:teuthology.orchestra.run.smithi146.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.4 MB/s | 249 kB 00:00 2024-04-25T15:17:35.493 INFO:teuthology.orchestra.run.smithi177.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-04-25T15:17:35.510 INFO:teuthology.orchestra.run.smithi177.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 167 kB/s | 36 kB 00:00 2024-04-25T15:17:35.516 INFO:teuthology.orchestra.run.smithi146.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 742 kB/s | 49 kB 00:00 2024-04-25T15:17:35.527 INFO:teuthology.orchestra.run.smithi177.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T15:17:35.561 INFO:teuthology.orchestra.run.smithi177.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T15:17:35.578 INFO:teuthology.orchestra.run.smithi177.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T15:17:35.603 INFO:teuthology.orchestra.run.smithi177.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-25T15:17:35.608 INFO:teuthology.orchestra.run.smithi146.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 180 kB/s | 16 kB 00:00 2024-04-25T15:17:35.620 INFO:teuthology.orchestra.run.smithi177.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-25T15:17:35.637 INFO:teuthology.orchestra.run.smithi177.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T15:17:35.654 INFO:teuthology.orchestra.run.smithi177.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 668 kB/s | 22 kB 00:00 2024-04-25T15:17:35.687 INFO:teuthology.orchestra.run.smithi177.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 17 MB/s | 863 kB 00:00 2024-04-25T15:17:35.758 INFO:teuthology.orchestra.run.smithi146.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.8 MB/s | 273 kB 00:00 2024-04-25T15:17:35.871 INFO:teuthology.orchestra.run.smithi177.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.5 MB/s | 548 kB 00:00 2024-04-25T15:17:35.896 INFO:teuthology.orchestra.run.smithi177.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-04-25T15:17:35.913 INFO:teuthology.orchestra.run.smithi177.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 464 kB/s | 19 kB 00:00 2024-04-25T15:17:35.968 INFO:teuthology.orchestra.run.smithi177.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-04-25T15:17:36.014 INFO:teuthology.orchestra.run.smithi177.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-04-25T15:17:36.056 INFO:teuthology.orchestra.run.smithi177.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-25T15:17:36.084 INFO:teuthology.orchestra.run.smithi146.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 2.3 MB/s | 762 kB 00:00 2024-04-25T15:17:36.123 INFO:teuthology.orchestra.run.smithi177.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-25T15:17:36.148 INFO:teuthology.orchestra.run.smithi177.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-25T15:17:36.190 INFO:teuthology.orchestra.run.smithi177.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-25T15:17:36.215 INFO:teuthology.orchestra.run.smithi177.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-25T15:17:36.240 INFO:teuthology.orchestra.run.smithi177.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 865 kB/s | 43 kB 00:00 2024-04-25T15:17:36.266 INFO:teuthology.orchestra.run.smithi177.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-04-25T15:17:36.300 INFO:teuthology.orchestra.run.smithi177.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 254 kB/s | 15 kB 00:00 2024-04-25T15:17:36.325 INFO:teuthology.orchestra.run.smithi177.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T15:17:36.358 INFO:teuthology.orchestra.run.smithi177.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-25T15:17:36.384 INFO:teuthology.orchestra.run.smithi177.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-04-25T15:17:36.409 INFO:teuthology.orchestra.run.smithi177.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-04-25T15:17:36.434 INFO:teuthology.orchestra.run.smithi177.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-25T15:17:36.467 INFO:teuthology.orchestra.run.smithi146.stdout:(37/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 7.7 MB/s | 43 MB 00:05 2024-04-25T15:17:36.485 INFO:teuthology.orchestra.run.smithi177.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 27 MB/s | 2.0 MB 00:00 2024-04-25T15:17:36.493 INFO:teuthology.orchestra.run.smithi146.stdout:(38/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 45 kB/s | 18 kB 00:00 2024-04-25T15:17:36.510 INFO:teuthology.orchestra.run.smithi177.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T15:17:36.535 INFO:teuthology.orchestra.run.smithi177.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-25T15:17:36.569 INFO:teuthology.orchestra.run.smithi177.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-25T15:17:36.594 INFO:teuthology.orchestra.run.smithi177.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-04-25T15:17:36.610 INFO:teuthology.orchestra.run.smithi146.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-04-25T15:17:36.619 INFO:teuthology.orchestra.run.smithi177.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-25T15:17:36.645 INFO:teuthology.orchestra.run.smithi177.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-04-25T15:17:36.660 INFO:teuthology.orchestra.run.smithi146.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-04-25T15:17:36.677 INFO:teuthology.orchestra.run.smithi146.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-25T15:17:36.695 INFO:teuthology.orchestra.run.smithi177.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 927 kB/s | 46 kB 00:00 2024-04-25T15:17:36.711 INFO:teuthology.orchestra.run.smithi146.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T15:17:36.728 INFO:teuthology.orchestra.run.smithi177.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-04-25T15:17:36.754 INFO:teuthology.orchestra.run.smithi177.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-25T15:17:36.769 INFO:teuthology.orchestra.run.smithi146.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-25T15:17:36.779 INFO:teuthology.orchestra.run.smithi177.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-25T15:17:36.794 INFO:teuthology.orchestra.run.smithi146.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-04-25T15:17:36.804 INFO:teuthology.orchestra.run.smithi177.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-25T15:17:36.820 INFO:teuthology.orchestra.run.smithi146.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T15:17:36.829 INFO:teuthology.orchestra.run.smithi177.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-25T15:17:36.845 INFO:teuthology.orchestra.run.smithi146.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T15:17:36.855 INFO:teuthology.orchestra.run.smithi177.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-25T15:17:36.870 INFO:teuthology.orchestra.run.smithi146.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T15:17:36.880 INFO:teuthology.orchestra.run.smithi177.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-25T15:17:36.904 INFO:teuthology.orchestra.run.smithi146.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-25T15:17:36.914 INFO:teuthology.orchestra.run.smithi177.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-04-25T15:17:36.940 INFO:teuthology.orchestra.run.smithi177.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-25T15:17:36.945 INFO:teuthology.orchestra.run.smithi146.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-25T15:17:36.964 INFO:teuthology.orchestra.run.smithi177.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-25T15:17:36.971 INFO:teuthology.orchestra.run.smithi146.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T15:17:36.990 INFO:teuthology.orchestra.run.smithi177.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-04-25T15:17:36.996 INFO:teuthology.orchestra.run.smithi146.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-25T15:17:37.015 INFO:teuthology.orchestra.run.smithi177.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T15:17:37.021 INFO:teuthology.orchestra.run.smithi146.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T15:17:37.040 INFO:teuthology.orchestra.run.smithi177.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-25T15:17:37.047 INFO:teuthology.orchestra.run.smithi146.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 661 kB/s | 33 kB 00:00 2024-04-25T15:17:37.066 INFO:teuthology.orchestra.run.smithi177.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T15:17:37.108 INFO:teuthology.orchestra.run.smithi177.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-04-25T15:17:37.134 INFO:teuthology.orchestra.run.smithi177.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-04-25T15:17:37.139 INFO:teuthology.orchestra.run.smithi146.stdout:(54/152): ceph-mgr-diskprediction-local-19.0.0- 3.2 MB/s | 7.4 MB 00:02 2024-04-25T15:17:37.161 INFO:teuthology.orchestra.run.smithi177.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T15:17:37.172 INFO:teuthology.orchestra.run.smithi146.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 148 kB/s | 18 kB 00:00 2024-04-25T15:17:37.200 INFO:teuthology.orchestra.run.smithi177.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-25T15:17:37.231 INFO:teuthology.orchestra.run.smithi146.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-25T15:17:37.248 INFO:teuthology.orchestra.run.smithi146.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 104 kB/s | 11 kB 00:00 2024-04-25T15:17:37.273 INFO:teuthology.orchestra.run.smithi146.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-25T15:17:37.290 INFO:teuthology.orchestra.run.smithi146.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T15:17:37.315 INFO:teuthology.orchestra.run.smithi146.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 662 kB 00:00 2024-04-25T15:17:37.340 INFO:teuthology.orchestra.run.smithi146.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-25T15:17:37.357 INFO:teuthology.orchestra.run.smithi146.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-25T15:17:37.374 INFO:teuthology.orchestra.run.smithi146.stdout:(63/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 442 kB/s | 25 kB 00:00 2024-04-25T15:17:37.399 INFO:teuthology.orchestra.run.smithi146.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-25T15:17:37.425 INFO:teuthology.orchestra.run.smithi146.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T15:17:37.442 INFO:teuthology.orchestra.run.smithi146.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 35 kB 00:00 2024-04-25T15:17:37.622 INFO:teuthology.orchestra.run.smithi146.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-04-25T15:17:37.659 INFO:teuthology.orchestra.run.smithi146.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 7.4 MB/s | 1.6 MB 00:00 2024-04-25T15:17:37.717 INFO:teuthology.orchestra.run.smithi146.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-25T15:17:37.746 INFO:teuthology.orchestra.run.smithi146.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 7.4 MB/s | 911 kB 00:00 2024-04-25T15:17:37.768 INFO:teuthology.orchestra.run.smithi146.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-25T15:17:37.827 INFO:teuthology.orchestra.run.smithi146.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-25T15:17:37.852 INFO:teuthology.orchestra.run.smithi146.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-04-25T15:17:37.944 INFO:teuthology.orchestra.run.smithi146.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 29 MB/s | 5.8 MB 00:00 2024-04-25T15:17:37.977 INFO:teuthology.orchestra.run.smithi146.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-25T15:17:37.984 INFO:teuthology.orchestra.run.smithi177.stdout:(148/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 3.9 MB/s | 3.1 MB 00:00 2024-04-25T15:17:38.042 INFO:teuthology.orchestra.run.smithi177.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T15:17:38.061 INFO:teuthology.orchestra.run.smithi146.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-04-25T15:17:38.086 INFO:teuthology.orchestra.run.smithi146.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-04-25T15:17:38.111 INFO:teuthology.orchestra.run.smithi146.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 196 kB/s | 26 kB 00:00 2024-04-25T15:17:38.136 INFO:teuthology.orchestra.run.smithi146.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 256 kB/s | 19 kB 00:00 2024-04-25T15:17:38.143 INFO:teuthology.orchestra.run.smithi177.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-25T15:17:38.162 INFO:teuthology.orchestra.run.smithi146.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 697 kB/s | 35 kB 00:00 2024-04-25T15:17:38.187 INFO:teuthology.orchestra.run.smithi146.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T15:17:38.212 INFO:teuthology.orchestra.run.smithi146.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 250 kB 00:00 2024-04-25T15:17:38.246 INFO:teuthology.orchestra.run.smithi146.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.8 MB/s | 222 kB 00:00 2024-04-25T15:17:38.271 INFO:teuthology.orchestra.run.smithi146.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-25T15:17:38.338 INFO:teuthology.orchestra.run.smithi146.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-25T15:17:38.363 INFO:teuthology.orchestra.run.smithi146.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 368 kB/s | 42 kB 00:00 2024-04-25T15:17:38.388 INFO:teuthology.orchestra.run.smithi146.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-04-25T15:17:38.414 INFO:teuthology.orchestra.run.smithi146.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T15:17:38.439 INFO:teuthology.orchestra.run.smithi146.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-04-25T15:17:38.464 INFO:teuthology.orchestra.run.smithi146.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-04-25T15:17:38.489 INFO:teuthology.orchestra.run.smithi146.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T15:17:38.515 INFO:teuthology.orchestra.run.smithi146.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-04-25T15:17:38.548 INFO:teuthology.orchestra.run.smithi146.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 783 kB/s | 44 kB 00:00 2024-04-25T15:17:38.582 INFO:teuthology.orchestra.run.smithi146.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.4 MB/s | 221 kB 00:00 2024-04-25T15:17:38.616 INFO:teuthology.orchestra.run.smithi146.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 566 kB/s | 36 kB 00:00 2024-04-25T15:17:38.649 INFO:teuthology.orchestra.run.smithi146.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 748 kB/s | 47 kB 00:00 2024-04-25T15:17:38.683 INFO:teuthology.orchestra.run.smithi146.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.7 MB/s | 305 kB 00:00 2024-04-25T15:17:38.725 INFO:teuthology.orchestra.run.smithi146.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 804 kB/s | 58 kB 00:00 2024-04-25T15:17:38.875 INFO:teuthology.orchestra.run.smithi146.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-25T15:17:38.900 INFO:teuthology.orchestra.run.smithi146.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 439 kB/s | 94 kB 00:00 2024-04-25T15:17:38.925 INFO:teuthology.orchestra.run.smithi146.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 322 kB/s | 64 kB 00:00 2024-04-25T15:17:38.951 INFO:teuthology.orchestra.run.smithi146.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-04-25T15:17:38.968 INFO:teuthology.orchestra.run.smithi146.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-04-25T15:17:38.968 INFO:teuthology.orchestra.run.smithi177.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 1.8 MB/s | 3.3 MB 00:01 2024-04-25T15:17:39.002 INFO:teuthology.orchestra.run.smithi146.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T15:17:39.485 INFO:teuthology.orchestra.run.smithi146.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-25T15:17:39.510 INFO:teuthology.orchestra.run.smithi146.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 38 kB/s | 19 kB 00:00 2024-04-25T15:17:39.602 INFO:teuthology.orchestra.run.smithi146.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 488 kB/s | 309 kB 00:00 2024-04-25T15:17:39.652 INFO:teuthology.orchestra.run.smithi146.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-25T15:17:39.744 INFO:teuthology.orchestra.run.smithi146.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 345 kB/s | 49 kB 00:00 2024-04-25T15:17:39.778 INFO:teuthology.orchestra.run.smithi146.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 542 kB/s | 67 kB 00:00 2024-04-25T15:17:39.945 INFO:teuthology.orchestra.run.smithi146.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-04-25T15:17:40.003 INFO:teuthology.orchestra.run.smithi146.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 583 kB/s | 151 kB 00:00 2024-04-25T15:17:40.037 INFO:teuthology.orchestra.run.smithi146.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-04-25T15:17:40.154 INFO:teuthology.orchestra.run.smithi146.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-04-25T15:17:40.179 INFO:teuthology.orchestra.run.smithi146.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 273 kB/s | 32 kB 00:00 2024-04-25T15:17:40.229 INFO:teuthology.orchestra.run.smithi146.stdout:(116/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 193 kB/s | 43 kB 00:00 2024-04-25T15:17:40.279 INFO:teuthology.orchestra.run.smithi146.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-04-25T15:17:40.397 INFO:teuthology.orchestra.run.smithi146.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 892 kB/s | 172 kB 00:00 2024-04-25T15:17:40.480 INFO:teuthology.orchestra.run.smithi146.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-04-25T15:17:40.505 INFO:teuthology.orchestra.run.smithi146.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.5 MB/s | 303 kB 00:00 2024-04-25T15:17:40.556 INFO:teuthology.orchestra.run.smithi146.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 222 kB/s | 35 kB 00:00 2024-04-25T15:17:40.698 INFO:teuthology.orchestra.run.smithi146.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 943 kB/s | 206 kB 00:00 2024-04-25T15:17:40.756 INFO:teuthology.orchestra.run.smithi146.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 824 kB/s | 144 kB 00:00 2024-04-25T15:17:40.815 INFO:teuthology.orchestra.run.smithi146.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-04-25T15:17:40.873 INFO:teuthology.orchestra.run.smithi146.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-04-25T15:17:40.924 INFO:teuthology.orchestra.run.smithi146.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 106 kB/s | 18 kB 00:00 2024-04-25T15:17:41.007 INFO:teuthology.orchestra.run.smithi146.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 121 kB/s | 23 kB 00:00 2024-04-25T15:17:41.032 INFO:teuthology.orchestra.run.smithi146.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 144 kB/s | 19 kB 00:00 2024-04-25T15:17:41.083 INFO:teuthology.orchestra.run.smithi146.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 146 kB/s | 20 kB 00:00 2024-04-25T15:17:41.225 INFO:teuthology.orchestra.run.smithi146.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-04-25T15:17:41.250 INFO:teuthology.orchestra.run.smithi146.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 240 kB/s | 46 kB 00:00 2024-04-25T15:17:41.300 INFO:teuthology.orchestra.run.smithi146.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 409 kB/s | 79 kB 00:00 2024-04-25T15:17:41.384 INFO:teuthology.orchestra.run.smithi146.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 363 kB/s | 58 kB 00:00 2024-04-25T15:17:41.443 INFO:teuthology.orchestra.run.smithi146.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-04-25T15:17:41.468 INFO:teuthology.orchestra.run.smithi146.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-04-25T15:17:41.527 INFO:teuthology.orchestra.run.smithi146.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 637 kB/s | 90 kB 00:00 2024-04-25T15:17:41.610 INFO:teuthology.orchestra.run.smithi146.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 170 kB/s | 28 kB 00:00 2024-04-25T15:17:41.669 INFO:teuthology.orchestra.run.smithi146.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 175 kB/s | 31 kB 00:00 2024-04-25T15:17:41.769 INFO:teuthology.orchestra.run.smithi146.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 375 kB/s | 59 kB 00:00 2024-04-25T15:17:41.794 INFO:teuthology.orchestra.run.smithi146.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 780 kB/s | 188 kB 00:00 2024-04-25T15:17:41.844 INFO:teuthology.orchestra.run.smithi146.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 205 kB/s | 36 kB 00:00 2024-04-25T15:17:41.936 INFO:teuthology.orchestra.run.smithi146.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 306 kB/s | 51 kB 00:00 2024-04-25T15:17:42.012 INFO:teuthology.orchestra.run.smithi146.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 630 kB/s | 90 kB 00:00 2024-04-25T15:17:42.045 INFO:teuthology.orchestra.run.smithi146.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-04-25T15:17:42.137 INFO:teuthology.orchestra.run.smithi146.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.1 MB/s | 427 kB 00:00 2024-04-25T15:17:42.154 INFO:teuthology.orchestra.run.smithi146.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 141 kB/s | 20 kB 00:00 2024-04-25T15:17:42.321 INFO:teuthology.orchestra.run.smithi146.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 763 kB/s | 192 kB 00:00 2024-04-25T15:17:42.404 INFO:teuthology.orchestra.run.smithi146.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-04-25T15:17:42.463 INFO:teuthology.orchestra.run.smithi146.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T15:17:42.546 INFO:teuthology.orchestra.run.smithi146.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T15:17:43.480 INFO:teuthology.orchestra.run.smithi146.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 2.7 MB/s | 3.1 MB 00:01 2024-04-25T15:17:43.688 INFO:teuthology.orchestra.run.smithi146.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 2.2 MB/s | 3.3 MB 00:01 2024-04-25T15:17:43.693 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:17:43.693 INFO:teuthology.orchestra.run.smithi146.stdout:Total 11 MB/s | 214 MB 00:20 2024-04-25T15:17:44.764 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-04-25T15:17:44.908 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-04-25T15:17:44.908 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-04-25T15:17:46.666 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-04-25T15:17:46.666 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-04-25T15:17:46.700 INFO:teuthology.orchestra.run.smithi177.stdout:(152/152): ceph-test-19.0.0-3291.g994e2f20.el9. 2.4 MB/s | 43 MB 00:17 2024-04-25T15:17:46.706 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:17:46.706 INFO:teuthology.orchestra.run.smithi177.stdout:Total 9.2 MB/s | 214 MB 00:23 2024-04-25T15:17:47.829 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-25T15:17:47.971 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-25T15:17:47.971 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-25T15:17:49.781 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-25T15:17:49.781 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-25T15:17:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-04-25T15:17:52.642 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:17:52.680 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:17:52.734 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:17:52.770 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:17:52.795 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:17:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:17:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:17:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:17:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:17:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:17:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:17:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:17:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:17:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:17:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:17:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:17:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:17:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:17:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:17:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:17:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:17:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:17:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:17:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:17:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:17:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:17:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:17:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:17:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:17:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:17:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:17:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:17:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:17:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:17:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:17:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:17:55.089 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:17:55.133 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:17:55.161 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:17:55.231 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:17:55.276 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:17:55.351 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:17:55.486 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:17:55.545 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:17:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-25T15:17:55.753 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:17:55.802 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:17:55.854 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:17:55.888 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:17:55.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:17:55.955 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:17:56.079 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:17:56.165 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:17:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:17:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:17:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:17:56.285 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:17:56.324 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:17:56.415 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:17:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:17:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:17:56.540 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:17:56.575 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:17:56.591 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:17:56.663 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:17:56.666 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:17:56.695 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:17:56.696 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:17:56.731 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:17:56.777 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:17:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:17:56.845 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:17:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:17:56.894 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:17:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:17:56.921 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:17:57.025 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:17:57.034 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:17:57.047 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:17:57.110 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:17:57.132 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:17:57.138 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:17:57.152 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:17:57.168 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:17:57.248 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:17:57.348 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:17:57.381 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:17:57.488 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:17:57.490 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:17:57.518 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:17:57.558 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:17:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:17:57.584 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:17:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:17:57.614 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:17:57.648 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:17:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:17:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:17:57.756 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:17:57.808 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:17:57.810 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:17:57.840 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:17:57.848 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:17:57.869 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:17:57.876 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:17:57.911 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:17:57.937 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:17:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:17:57.989 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:17:57.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:17:58.019 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:17:58.021 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:17:58.045 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:17:58.057 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:17:58.079 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:17:58.121 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:17:58.133 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:17:58.154 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:17:58.167 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:17:58.211 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:17:58.240 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:17:58.242 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:17:58.284 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:17:58.309 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:17:58.324 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:17:58.355 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:17:58.371 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:17:58.428 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:17:58.470 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:17:58.526 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:17:58.562 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:17:58.632 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:17:59.155 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:17:59.244 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:17:59.246 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:17:59.273 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:17:59.290 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:17:59.316 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:17:59.323 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:17:59.354 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:17:59.396 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:17:59.421 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:17:59.497 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:17:59.556 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:17:59.584 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:17:59.607 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:17:59.625 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:17:59.662 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:17:59.735 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:17:59.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:17:59.803 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:17:59.805 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:17:59.831 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:17:59.849 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:17:59.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:17:59.909 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:17:59.928 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:17:59.970 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:18:00.105 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:18:00.151 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:18:00.182 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:18:00.223 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:18:00.317 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:18:00.348 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:18:00.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:18:00.422 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:18:00.464 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:18:00.538 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:18:00.551 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:18:00.671 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:18:00.691 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:18:00.730 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:18:00.758 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:18:00.792 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:18:00.831 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:18:00.871 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:18:00.900 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:18:00.929 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:18:00.960 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:18:01.002 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:18:01.036 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:18:01.123 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:18:01.167 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:18:01.198 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:18:01.245 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:18:01.344 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:18:01.399 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:18:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:18:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:18:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:18:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:18:02.121 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:18:02.172 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:18:02.205 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:18:02.304 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:18:02.333 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:18:02.372 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:18:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:18:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:18:02.459 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:18:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:18:02.491 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:18:02.524 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:18:02.678 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:18:02.705 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:18:02.785 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:18:03.017 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:18:03.044 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:18:03.106 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:18:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:18:03.423 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:18:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:18:04.749 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:18:04.851 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:18:04.879 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:18:04.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:18:05.241 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:18:05.292 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:18:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:18:05.382 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:18:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:18:05.414 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:18:05.450 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:18:05.937 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:18:05.970 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:18:05.972 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:18:06.012 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:18:06.041 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:18:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:18:06.084 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:18:06.084 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:18:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:06.105 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:18:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:18:08.572 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:18:08.610 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:18:09.168 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:18:09.214 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:18:09.305 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:18:09.305 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:18:09.305 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:18:09.305 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:09.331 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:18:17.745 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:18:17.745 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-04-25T15:18:17.745 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-04-25T15:18:17.746 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-04-25T15:18:17.746 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-04-25T15:18:17.746 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-04-25T15:18:17.746 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-04-25T15:18:17.746 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-04-25T15:18:17.746 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:17.925 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:18:17.940 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:18:17.940 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:17.941 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:18:17.941 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:19.503 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:18:19.529 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:18:19.529 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:19.529 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:18:19.529 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:18:19.530 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:18:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:19.753 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:18:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:18:19.776 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:19.776 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:18:19.776 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:18:19.776 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:18:19.776 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:20.246 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:18:20.268 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:18:20.268 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:20.268 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:18:20.268 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:18:20.269 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:18:20.269 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:18:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:18:20.488 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:18:20.675 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:18:20.721 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:18:20.800 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-04-25T15:18:20.883 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:20.888 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:18:21.036 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:18:21.070 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:18:21.079 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:18:21.086 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:18:21.087 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:21.087 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:18:21.087 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:18:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:18:21.599 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:18:21.629 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:18:21.661 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:18:21.723 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:18:21.761 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:18:22.648 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:18:22.674 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:18:22.675 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:22.675 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:18:22.675 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:18:22.675 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:18:22.675 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:18:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:18:22.907 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:18:22.930 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:18:22.931 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:22.931 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:18:22.931 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:18:22.931 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:18:22.931 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:23.383 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:18:23.405 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:18:23.406 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:23.406 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:18:23.406 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:18:23.406 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:18:23.406 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:23.521 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:18:23.577 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:18:23.633 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:18:23.814 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:18:23.816 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:18:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:18:23.866 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:18:23.935 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:18:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:18:24.025 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:18:24.104 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:18:24.139 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:18:24.157 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:18:24.162 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:18:24.162 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:24.163 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:18:24.163 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:18:24.163 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:18:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:24.192 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:18:24.215 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:18:24.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:18:24.226 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:18:24.252 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:18:24.325 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:18:24.372 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:18:24.704 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:18:24.734 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:18:24.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:18:24.836 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:18:24.874 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:18:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:18:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:18:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:18:25.374 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:25.374 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:18:25.374 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:18:25.375 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:18:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:18:25.440 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:18:25.440 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:25.440 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:18:25.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:25.485 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:18:25.508 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:18:25.508 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:25.508 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:18:25.508 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:18:25.509 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:18:25.509 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:18:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:18:25.842 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:25.842 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:18:25.842 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:18:25.842 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:18:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:25.910 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:18:25.943 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:18:26.909 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:18:26.934 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:18:27.037 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:18:27.201 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:18:27.228 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:18:27.252 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:18:27.252 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:27.252 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:18:27.252 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:18:27.252 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:18:27.252 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:27.295 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:18:27.307 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:18:27.407 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:18:27.460 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:18:28.396 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:18:28.432 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:18:28.455 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:18:28.455 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:28.455 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:18:28.455 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:18:28.455 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:18:28.455 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:28.499 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:18:28.521 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:18:28.521 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:28.521 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:18:28.521 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:28.566 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:18:28.589 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:18:28.589 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:28.590 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:18:28.590 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:18:28.590 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:18:28.590 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:28.900 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:18:28.923 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:18:28.923 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:18:28.923 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:18:28.923 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:18:28.924 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:18:28.924 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:30.131 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:18:30.173 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:18:30.213 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:18:30.245 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:18:30.273 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:18:30.459 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:18:30.510 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:18:30.540 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:18:31.120 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:18:31.154 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:18:31.174 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:18:32.020 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:18:32.109 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:18:32.149 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:18:32.231 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:18:32.272 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:18:32.272 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:18:32.338 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:18:32.364 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:18:32.364 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:18:32.440 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:18:32.503 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:18:33.171 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:18:33.214 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:18:33.261 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:18:33.293 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:18:33.320 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:18:33.505 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:18:33.558 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:18:33.588 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:18:34.196 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:18:34.227 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:18:34.248 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:18:35.075 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:18:35.166 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:18:35.207 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:18:35.289 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:18:35.322 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:18:35.322 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:18:35.404 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:18:35.437 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:18:35.438 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:18:35.556 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:18:35.619 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:18:38.225 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:18:38.226 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:18:38.227 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:18:38.239 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:18:38.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:18:38.241 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:18:38.242 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:18:38.243 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:18:38.244 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:18:38.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:18:38.246 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:18:38.247 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:18:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:18:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:18:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:18:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:18:38.248 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:18:40.386 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:18:40.387 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.388 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:18:40.389 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:18:40.390 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:18:40.391 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:18:40.392 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:18:40.393 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:18:40.394 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:18:40.395 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:18:40.396 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:18:40.397 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:18:40.398 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:18:40.398 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:18:40.398 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:18:40.398 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-04-25T15:18:40.796 DEBUG:teuthology.parallel:result is None 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:18:41.494 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:18:41.495 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:18:41.496 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:18:41.508 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:18:41.509 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:18:41.510 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:18:41.511 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:18:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:18:41.513 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:18:41.514 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:18:41.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:18:41.516 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:18:41.517 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:18:41.517 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:18:41.517 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:18:41.517 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:18:41.517 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:18:43.543 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.544 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.545 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:18:43.546 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:18:43.547 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:18:43.548 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:18:43.549 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:18:43.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:18:43.551 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:18:43.552 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:18:43.553 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:18:43.554 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:18:43.555 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:18:43.555 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:43.555 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-25T15:18:43.937 DEBUG:teuthology.parallel:result is None 2024-04-25T15:18:43.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:18:44.165 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:18:44.191 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:18:44.191 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:18:44.192 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:18:44.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:18:44.418 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:18:44.439 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:18:44.440 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:18:44.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:18:44.442 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T15:18:44.442 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:44.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:18:44.477 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:44.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:18:44.507 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T15:18:44.507 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:44.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:18:44.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:18:44.610 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:44.610 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:18:44.634 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:18:44.697 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T15:18:44.698 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:44.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:18:44.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:18:44.791 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:44.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:18:44.816 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:18:44.881 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T15:18:44.881 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:44.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:18:44.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:18:44.971 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:44.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:18:44.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:18:45.060 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T15:18:45.166 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T15:18:45.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-04-25T15:18:45.169 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-04-25T15:18:45.193 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T15:18:45.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T15:18:45.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T15:18:45.259 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T15:18:45.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T15:18:45.300 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T15:18:45.331 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T15:18:45.332 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T15:18:45.332 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T15:18:45.332 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:45.332 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T15:18:45.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T15:18:45.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T15:18:45.523 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:45.523 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T15:18:45.538 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T15:18:45.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T15:18:45.691 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T15:18:45.692 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T15:18:45.692 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673201', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1880, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'subset': '178/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQcOlVxd1/CY5+GNNxG/GW2bWEGfxv1T+F972WipdFff3Uoq5q2kf5snnF+Wr0YvTlLWQb0yOwELfzjQyBMzQ8=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3cCg9gTCONg9DJ8ATtl2/Liuyg4DoCd5g0p6RrKYZZBcBCL5Xf3J/DMVLliAWXV3y76EzJas7ENCk58VwXd18='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_quiesce']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_10:00:23', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T15:18:45.692 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T15:18:45.695 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T15:18:45.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T15:18:45.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T15:18:45.805 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:45.805 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:18:45.826 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:18:45.827 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-04-25T15:18:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:18:45.881 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-25 15:18:32.875594442 +0000 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-25 15:16:43.067066668 +0000 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-25 15:16:43.067066668 +0000 2024-04-25T15:18:45.882 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-25 15:16:43.067066668 +0000 2024-04-25T15:18:45.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:18:45.943 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-25T15:18:45.944 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-25T15:18:45.944 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000270176 s, 1.9 MB/s 2024-04-25T15:18:45.945 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:18:46.000 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-04-25T15:18:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:18:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-25 15:18:32.875594442 +0000 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-25 15:16:43.413060711 +0000 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-25 15:16:43.413060711 +0000 2024-04-25T15:18:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-25 15:16:43.413060711 +0000 2024-04-25T15:18:46.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:18:46.118 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-25T15:18:46.118 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-25T15:18:46.118 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000157226 s, 3.3 MB/s 2024-04-25T15:18:46.119 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:18:46.175 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-04-25T15:18:46.230 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:18:46.230 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:46.230 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T15:18:46.231 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:46.231 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:46.231 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-25 15:18:32.875594442 +0000 2024-04-25T15:18:46.231 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-25 15:16:43.754054840 +0000 2024-04-25T15:18:46.231 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-25 15:16:43.754054840 +0000 2024-04-25T15:18:46.231 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-25 15:16:43.754054840 +0000 2024-04-25T15:18:46.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:18:46.293 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-25T15:18:46.293 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-25T15:18:46.294 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000181806 s, 2.8 MB/s 2024-04-25T15:18:46.295 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:18:46.351 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-04-25T15:18:46.406 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-25 15:18:32.876594426 +0000 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-25 15:16:44.105048798 +0000 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-25 15:16:44.105048798 +0000 2024-04-25T15:18:46.407 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-25 15:16:44.105048798 +0000 2024-04-25T15:18:46.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:18:46.469 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-25T15:18:46.469 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-25T15:18:46.469 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000183191 s, 2.8 MB/s 2024-04-25T15:18:46.471 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:18:46.526 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:18:46.526 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:46.526 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:18:46.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:18:46.542 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-25 15:18:35.977201203 +0000 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-25 15:16:43.055894927 +0000 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-25 15:16:43.055894927 +0000 2024-04-25T15:18:46.597 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-25 15:16:43.055894927 +0000 2024-04-25T15:18:46.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:18:46.660 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-25T15:18:46.660 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-25T15:18:46.660 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000187707 s, 2.7 MB/s 2024-04-25T15:18:46.662 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:18:46.718 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-25 15:18:35.978201176 +0000 2024-04-25T15:18:46.773 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-25 15:16:43.402885118 +0000 2024-04-25T15:18:46.774 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-25 15:16:43.402885118 +0000 2024-04-25T15:18:46.774 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-25 15:16:43.402885118 +0000 2024-04-25T15:18:46.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:18:46.835 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-25T15:18:46.835 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-25T15:18:46.835 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000187698 s, 2.7 MB/s 2024-04-25T15:18:46.836 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:18:46.892 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-25T15:18:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-25 15:18:35.978201176 +0000 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-25 15:16:43.739875592 +0000 2024-04-25T15:18:46.948 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-25 15:16:43.739875592 +0000 2024-04-25T15:18:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-25 15:16:43.739875592 +0000 2024-04-25T15:18:46.949 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:18:47.010 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-25T15:18:47.010 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-25T15:18:47.011 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000180315 s, 2.8 MB/s 2024-04-25T15:18:47.012 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:18:47.067 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-25T15:18:47.122 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-25 15:18:35.978201176 +0000 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-25 15:16:44.073866151 +0000 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-25 15:16:44.073866151 +0000 2024-04-25T15:18:47.123 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-25 15:16:44.073866151 +0000 2024-04-25T15:18:47.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:18:47.184 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-25T15:18:47.185 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-25T15:18:47.185 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000226985 s, 2.3 MB/s 2024-04-25T15:18:47.186 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:18:47.241 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:18:47.241 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T15:18:47.241 INFO:tasks.ceph:Generating config... 2024-04-25T15:18:47.243 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T15:18:47.243 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T15:18:47.243 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T15:18:47.243 INFO:tasks.ceph:[client] pid file = 2024-04-25T15:18:47.243 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T15:18:47.243 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T15:18:47.243 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T15:18:47.244 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T15:18:47.244 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T15:18:47.244 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T15:18:47.244 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T15:18:47.244 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T15:18:47.245 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T15:18:47.245 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T15:18:47.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T15:18:47.313 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T15:18:47.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T15:18:47.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:18:47.393 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.146'), ('mon.b', '172.21.15.177'), ('mon.c', '[v2:172.21.15.177:3301,v1:172.21.15.177:6790]')] 2024-04-25T15:18:47.394 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.146,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T15:18:47.395 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:47.396 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T15:18:47.450 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.146 --add b 172.21.15.177 --addv c '[v2:172.21.15.177:3301,v1:172.21.15.177:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: generated fsid c6e9a678-3722-48f4-a297-70f509c13e29 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:setting min_mon_release = quincy 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 0 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:fsid c6e9a678-3722-48f4-a297-70f509c13e29 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:last_changed 2024-04-25T15:18:47.545979+0000 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:created 2024-04-25T15:18:47.545979+0000 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:min_mon_release 17 (quincy) 2024-04-25T15:18:47.546 INFO:teuthology.orchestra.run.smithi146.stdout:election_strategy: 1 2024-04-25T15:18:47.547 INFO:teuthology.orchestra.run.smithi146.stdout:0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-25T15:18:47.547 INFO:teuthology.orchestra.run.smithi146.stdout:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-04-25T15:18:47.547 INFO:teuthology.orchestra.run.smithi146.stdout:2: [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] mon.c 2024-04-25T15:18:47.547 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T15:18:47.548 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T15:18:47.563 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c6e9a678-3722-48f4-a297-70f509c13e29... 2024-04-25T15:18:47.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T15:18:47.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T15:18:47.644 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-04-25T15:18:47.644 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-04-25T15:18:47.644 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T15:18:47.644 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.645 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-04-25T15:18:47.646 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.146,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = c6e9a678-3722-48f4-a297-70f509c13e29 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-04-25T15:18:47.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-04-25T15:18:47.648 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-04-25T15:18:47.649 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 20 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 20 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 10 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-04-25T15:18:47.650 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-04-25T15:18:47.651 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T15:18:47.652 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: mon down mkfs grace = 300 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T15:18:47.653 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-04-25T15:18:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-04-25T15:18:47.655 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.661 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.662 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-04-25T15:18:47.663 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-04-25T15:18:47.664 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.146,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = c6e9a678-3722-48f4-a297-70f509c13e29 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-04-25T15:18:47.665 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-04-25T15:18:47.667 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.668 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 20 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 20 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-04-25T15:18:47.669 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-04-25T15:18:47.670 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.671 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: mon down mkfs grace = 300 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-25T15:18:47.672 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-04-25T15:18:47.673 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-04-25T15:18:47.674 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-04-25T15:18:47.675 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-04-25T15:18:47.675 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-04-25T15:18:47.675 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-04-25T15:18:47.675 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-04-25T15:18:47.676 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T15:18:47.676 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T15:18:47.759 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T15:18:47.760 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:47.760 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T15:18:47.775 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:47.775 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T15:18:47.829 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:18:47.830 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:47.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T15:18:47.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:18:47.903 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:18:47.903 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:47.957 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:18:47.957 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:47.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T15:18:47.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:18:47.990 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:18:47.990 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:48.045 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T15:18:48.045 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T15:18:48.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T15:18:48.102 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T15:18:48.106 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T15:18:48.181 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T15:18:48.184 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T15:18:48.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T15:18:48.242 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T15:18:48.245 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T15:18:48.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T15:18:48.368 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T15:18:48.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T15:18:48.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T15:18:48.451 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T15:18:48.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T15:18:48.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T15:18:48.575 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T15:18:48.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T15:18:48.601 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T15:18:48.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T15:18:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T15:18:48.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-25T15:18:48.710 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-25T15:18:48.723 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T15:18:48.771 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T15:18:48.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-25T15:18:48.831 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-25T15:18:48.843 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T15:18:48.843 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T15:18:48.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T15:18:48.867 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:18:48.867 INFO:tasks.ceph:role: osd.0 2024-04-25T15:18:48.867 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:18:48.867 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T15:18:48.938 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:18:48.938 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:18:48.938 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:18:48.939 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:18:49.455 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-04-25T15:18:49.457 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-04-25T15:18:49.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T15:18:49.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T15:18:49.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T15:18:49.677 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:18:49.677 INFO:tasks.ceph:role: osd.1 2024-04-25T15:18:49.677 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:18:49.677 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T15:18:49.748 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:18:49.749 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:18:49.750 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:18:50.217 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-04-25T15:18:50.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-04-25T15:18:50.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T15:18:50.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T15:18:50.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T15:18:50.403 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:18:50.403 INFO:tasks.ceph:role: osd.2 2024-04-25T15:18:50.403 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:18:50.403 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:18:50.474 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:18:50.475 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:18:50.475 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:18:50.475 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:18:50.958 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-04-25T15:18:50.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-04-25T15:18:50.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T15:18:51.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T15:18:51.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T15:18:51.174 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:18:51.174 INFO:tasks.ceph:role: osd.3 2024-04-25T15:18:51.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:18:51.174 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:18:51.245 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:18:51.736 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-04-25T15:18:51.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-04-25T15:18:51.738 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T15:18:51.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T15:18:51.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:52.009+0000 7ffbc4f44640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T15:18:52.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:52.009+0000 7ffbc4f44640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T15:18:52.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:52.009+0000 7ffbc4f44640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:18:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:52.009+0000 7ffbc4f44640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:18:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:52.009+0000 7ffbc4f44640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:18:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:52.010+0000 7ffbc4f44640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T15:18:54.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T15:18:54.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:54.992+0000 7f08fe20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T15:18:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:54.993+0000 7f08fe20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T15:18:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:54.993+0000 7f08fe20d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:18:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:54.993+0000 7f08fe20d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:18:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:54.993+0000 7f08fe20d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:18:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:54.993+0000 7f08fe20d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T15:18:57.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T15:18:57.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:18:57.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:57.965+0000 7f7449617640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T15:18:57.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:57.965+0000 7f7449617640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T15:18:57.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:57.965+0000 7f7449617640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:18:57.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:57.965+0000 7f7449617640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:18:57.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:57.965+0000 7f7449617640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:18:57.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:18:57.966+0000 7f7449617640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T15:19:00.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T15:19:00.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:00.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:00.947+0000 7f06c3a9a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T15:19:00.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:00.947+0000 7f06c3a9a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T15:19:00.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:00.947+0000 7f06c3a9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:19:00.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:00.947+0000 7f06c3a9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:19:00.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:00.947+0000 7f06c3a9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:19:00.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:00.948+0000 7f06c3a9a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T15:19:03.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T15:19:03.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T15:19:03.858 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:19:03.858 INFO:tasks.ceph:role: osd.4 2024-04-25T15:19:03.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:19:03.858 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T15:19:03.933 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:19:03.934 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:19:04.429 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-25T15:19:04.431 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-25T15:19:04.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T15:19:04.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T15:19:04.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T15:19:04.668 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:19:04.668 INFO:tasks.ceph:role: osd.5 2024-04-25T15:19:04.668 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:19:04.668 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T15:19:04.740 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:19:04.740 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:19:04.740 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:19:04.740 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:19:04.741 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:19:04.741 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:19:04.741 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:19:04.741 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:19:04.741 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:19:04.741 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:19:05.185 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-25T15:19:05.187 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-25T15:19:05.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T15:19:05.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T15:19:05.302 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T15:19:05.366 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:19:05.366 INFO:tasks.ceph:role: osd.6 2024-04-25T15:19:05.366 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:19:05.366 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T15:19:05.437 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:19:05.438 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:19:05.921 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-25T15:19:05.924 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-25T15:19:05.924 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T15:19:06.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T15:19:06.083 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T15:19:06.147 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:19:06.147 INFO:tasks.ceph:role: osd.7 2024-04-25T15:19:06.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:19:06.148 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T15:19:06.218 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:19:06.219 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:19:06.717 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-04-25T15:19:06.719 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-04-25T15:19:06.719 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T15:19:06.833 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T15:19:06.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:06.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:06.965+0000 7f7384419640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T15:19:06.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:06.965+0000 7f7384419640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T15:19:06.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:06.965+0000 7f7384419640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:19:06.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:06.965+0000 7f7384419640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:19:06.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:06.965+0000 7f7384419640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:19:06.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:06.966+0000 7f7384419640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T15:19:09.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T15:19:09.846 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:09.948 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:09.946+0000 7f291f416640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T15:19:09.948 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:09.947+0000 7f291f416640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T15:19:09.949 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:09.947+0000 7f291f416640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:19:09.949 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:09.947+0000 7f291f416640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:19:09.949 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:09.947+0000 7f291f416640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:19:09.949 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:09.947+0000 7f291f416640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T15:19:12.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T15:19:12.818 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:12.916 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:12.914+0000 7fc8e3a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T15:19:12.916 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:12.915+0000 7fc8e3a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T15:19:12.916 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:12.915+0000 7fc8e3a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:19:12.916 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:12.915+0000 7fc8e3a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:19:12.916 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:12.915+0000 7fc8e3a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:19:12.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:12.915+0000 7fc8e3a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T15:19:15.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:19:15.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:19:15.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T15:19:15.834 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:15.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:15.927+0000 7f295f816640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T15:19:15.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:15.927+0000 7f295f816640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T15:19:15.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:15.927+0000 7f295f816640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:19:15.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:15.927+0000 7f295f816640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:19:15.928 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:15.927+0000 7f295f816640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:19:15.929 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:15.928+0000 7f295f816640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T15:19:18.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T15:19:18.805 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T15:19:18.805 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:18.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T15:19:18.834 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:18.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T15:19:18.902 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:18.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T15:19:18.968 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:18.968 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T15:19:19.033 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:19.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T15:19:19.097 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:19.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T15:19:19.162 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:19.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T15:19:19.227 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T15:19:19.255 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T15:19:19.325 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.325 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T15:19:19.389 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T15:19:19.455 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T15:19:19.520 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T15:19:19.584 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T15:19:19.647 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:19.648 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T15:19:19.663 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:19.663 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-25T15:19:19.719 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.719 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T15:19:19.734 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:19.737 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-25T15:19:19.789 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T15:19:19.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T15:19:19.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T15:19:19.832 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-04-25T15:19:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmmk8XBhAASX5Dc72zChEYixIh/iiawg== 2024-04-25T15:19:19.832 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-04-25T15:19:19.832 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpm/Ux1DhAA3DlkdKegZpZxlaZPFjodBw== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmtDPsFRAAh1HLdvyYLR78fokIKaekrA== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBcdCpmkFKVABAA+xGQMOacMJi18fh2p3rJFA== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBedCpm78w0OxAAGf4e1jZ0aAxnX/HBagE3yg== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBhdCpm2VCWORAAUEOB2XJBAbm1STvjkbjFNw== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBkdCpmfHmAOBAApJr+ddhMkB3As5PU2KALaQ== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmiMrSChAA/AOecro7QL+ypBbUDiwShQ== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmR1PoGhAAw/gk2lg0yvzUcOUZvwo6jg== 2024-04-25T15:19:19.833 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpm3EVMIhAAddbxnaiTto+gXb0+LGbRKA== 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBqdCpmtoKbORAA63fHSceN+QJCRCuFLZKlqA== 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBtdCpmIkKAOBAAN4te1g8Ioao/xQtjvw4vSg== 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBwdCpm90+YNhAApJ3X6R6BJC2VXiJ+RCl+TA== 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBzdCpmACtMNxAAFvU4+bypsr14tXk8suRhEQ== 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmlczBJhAApZj6jMADYxBHeBvhaprLDg== 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout:[client.2] 2024-04-25T15:19:19.834 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmhV1cKhAAdJQ+5g5RjIlilhKuxLcyiA== 2024-04-25T15:19:19.835 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-04-25T15:19:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmy/P3LRAAyZkQ/mpIZ9/H8m7/scVDKA== 2024-04-25T15:19:19.835 INFO:teuthology.orchestra.run.smithi146.stdout:[client.3] 2024-04-25T15:19:19.835 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmR6yLMRAAhFRT/TgCAnREAuUr53J1BA== 2024-04-25T15:19:19.851 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2024-04-25T15:19:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmmk8XBhAASX5Dc72zChEYixIh/iiawg== 2024-04-25T15:19:19.851 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-04-25T15:19:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpm/Ux1DhAA3DlkdKegZpZxlaZPFjodBw== 2024-04-25T15:19:19.851 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-04-25T15:19:19.851 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmtDPsFRAAh1HLdvyYLR78fokIKaekrA== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBcdCpmkFKVABAA+xGQMOacMJi18fh2p3rJFA== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBedCpm78w0OxAAGf4e1jZ0aAxnX/HBagE3yg== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBhdCpm2VCWORAAUEOB2XJBAbm1STvjkbjFNw== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBkdCpmfHmAOBAApJr+ddhMkB3As5PU2KALaQ== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmiMrSChAA/AOecro7QL+ypBbUDiwShQ== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmR1PoGhAAw/gk2lg0yvzUcOUZvwo6jg== 2024-04-25T15:19:19.852 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpm3EVMIhAAddbxnaiTto+gXb0+LGbRKA== 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBqdCpmtoKbORAA63fHSceN+QJCRCuFLZKlqA== 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBtdCpmIkKAOBAAN4te1g8Ioao/xQtjvw4vSg== 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBwdCpm90+YNhAApJ3X6R6BJC2VXiJ+RCl+TA== 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBzdCpmACtMNxAAFvU4+bypsr14tXk8suRhEQ== 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmlczBJhAApZj6jMADYxBHeBvhaprLDg== 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout:[client.2] 2024-04-25T15:19:19.853 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmhV1cKhAAdJQ+5g5RjIlilhKuxLcyiA== 2024-04-25T15:19:19.855 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-04-25T15:19:19.855 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmy/P3LRAAyZkQ/mpIZ9/H8m7/scVDKA== 2024-04-25T15:19:19.855 INFO:teuthology.orchestra.run.smithi177.stdout:[client.3] 2024-04-25T15:19:19.855 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYdCpmR6yLMRAAhFRT/TgCAnREAuUr53J1BA== 2024-04-25T15:19:19.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:19:19.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:19:19.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:19.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.143 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.145 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.266 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.324 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:19:20.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:19:20.385 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:19:20.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.514 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:19:20.758 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.760 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:20.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:19:21.011 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T15:19:21.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T15:19:21.035 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:19:21.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T15:19:21.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T15:19:21.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:19:21.294 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T15:19:21.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T15:19:21.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:19:21.486 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T15:19:21.509 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:21.511 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:19:21.564 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T15:19:21.564 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T15:19:21.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T15:19:21.567 INFO:tasks.ceph.mon.a:Started 2024-04-25T15:19:21.567 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T15:19:21.567 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T15:19:21.607 INFO:tasks.ceph.mon.b:Started 2024-04-25T15:19:21.607 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T15:19:21.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T15:19:21.609 INFO:tasks.ceph.mon.c:Started 2024-04-25T15:19:21.609 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T15:19:21.609 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T15:19:21.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T15:19:21.611 INFO:tasks.ceph.mgr.y:Started 2024-04-25T15:19:21.612 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T15:19:21.612 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T15:19:21.614 INFO:tasks.ceph.mgr.x:Started 2024-04-25T15:19:21.614 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T15:19:21.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T15:19:21.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.723+0000 7fe0a71c5640 1 Processor -- start 2024-04-25T15:19:21.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.723+0000 7fe0a71c5640 1 -- start start 2024-04-25T15:19:21.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a71c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe0a0169190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a71c5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a00b2d90 con 0x7fe0a00b3940 2024-04-25T15:19:21.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a71c5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a00b3050 con 0x7fe0a00d2e90 2024-04-25T15:19:21.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a71c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a00b2b60 con 0x7fe0a00b3d50 2024-04-25T15:19:21.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a4f61640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe0a00b3940 0x7fe0a00d2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34942/0 (socket says 172.21.15.146:34942) 2024-04-25T15:19:21.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a4f61640 1 -- 172.21.15.146:0/905154222 learned_addr learned my addr 172.21.15.146:0/905154222 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:21.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a5762640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe0a0169190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a5762640 1 -- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 msgr2=0x7fe0a0169190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T15:19:21.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a5762640 1 -- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 msgr2=0x7fe0a0169190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:19:21.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a5762640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe0a0169190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:19:21.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.725+0000 7fe0a5762640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe0a0169190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:19:21.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.888+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112291376 0 0) 0x7fe0a00b2d90 con 0x7fe0a00b3940 2024-04-25T15:19:21.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.888+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe088003520 con 0x7fe0a00b3940 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931556735 0 0) 0x7fe0a00b3050 con 0x7fe0a00d2e90 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a00b2d90 con 0x7fe0a00d2e90 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2953726389 0 0) 0x7fe0a00b2d90 con 0x7fe0a00d2e90 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 msgr2=0x7fe0a0169190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe0a0169190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 >> v1:172.21.15.146:6789/0 conn(0x7fe0a00b3940 legacy=0x7fe0a00d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a00b3330 con 0x7fe0a00d2e90 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fe094001670 con 0x7fe0a00d2e90 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 >> v1:172.21.15.177:6789/0 conn(0x7fe0a00d2e90 legacy=0x7fe0a0165a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe088003b40 0x7fe088003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe088004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe08800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe0880054d0 con 0x7fe088004800 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe088005680 con 0x7fe088003b40 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe08800bf80 con 0x7fe0a00b3d50 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09ffff640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe088004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe0a4f61640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe088003b40 0x7fe088003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe0a5762640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe08800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09ffff640 1 -- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 msgr2=0x7fe08800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09ffff640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a00b3d50 0x7fe08800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09ffff640 1 -- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe088003b40 msgr2=0x7fe088003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09ffff640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe088003b40 0x7fe088003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.889+0000 7fe09ffff640 1 -- 172.21.15.146:0/905154222 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a00b2d90 con 0x7fe088004800 2024-04-25T15:19:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.890+0000 7fe09ffff640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe088004c10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00d38b0 tx=0x7fe088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136b8612a2b08e6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.892+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094036070 con 0x7fe088004800 2024-04-25T15:19:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.892+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe094031620 con 0x7fe088004800 2024-04-25T15:19:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.892+0000 7fe09f7fe640 1 -- 172.21.15.146:0/905154222 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094007bb0 con 0x7fe088004800 2024-04-25T15:19:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a71c5640 1 -- 172.21.15.146:0/905154222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 msgr2=0x7fe088004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a71c5640 1 --2- 172.21.15.146:0/905154222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe088004c10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00d38b0 tx=0x7fe088005830 comp rx=0 tx=0).stop 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a5762640 1 -- 172.21.15.146:0/905154222 reap_dead start 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a71c5640 1 -- 172.21.15.146:0/905154222 shutdown_connections 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a71c5640 1 -- 172.21.15.146:0/905154222 >> 172.21.15.146:0/905154222 conn(0x7fe0a00b7490 msgr2=0x7fe0a00b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a71c5640 1 -- 172.21.15.146:0/905154222 shutdown_connections 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.893+0000 7fe0a71c5640 1 -- 172.21.15.146:0/905154222 wait complete. 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.895+0000 7fe0a71c5640 1 Processor -- start 2024-04-25T15:19:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.895+0000 7fe0a71c5640 1 -- start start 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.896+0000 7fe0a71c5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe0a00c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.896+0000 7fe0a71c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0a0151d60 0x7fe0a01e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.896+0000 7fe0a71c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a01e6bf0 0x7fe0a00b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.896+0000 7fe0a71c5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a009b7b0 con 0x7fe088004800 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.896+0000 7fe0a71c5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a009b630 con 0x7fe0a0151d60 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.896+0000 7fe0a71c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a00cf660 con 0x7fe0a01e6bf0 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe0a00c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe0a00c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45320/0 (socket says 172.21.15.146:45320) 2024-04-25T15:19:21.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 -- 172.21.15.146:0/3806367574 learned_addr learned my addr 172.21.15.146:0/3806367574 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe09ffff640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0a0151d60 0x7fe0a01e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a5762640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a01e6bf0 0x7fe0a00b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 -- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a01e6bf0 msgr2=0x7fe0a00b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a01e6bf0 0x7fe0a00b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 -- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0a0151d60 msgr2=0x7fe0a01e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0a0151d60 0x7fe0a01e66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 -- 172.21.15.146:0/3806367574 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a01e7000 con 0x7fe088004800 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a5762640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a01e6bf0 0x7fe0a00b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a4f61640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe0a00c7d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe090007840 tx=0x7fe0900018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09000f600 con 0x7fe088004800 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe090018070 con 0x7fe088004800 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a00c82c0 con 0x7fe088004800 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09000f9a0 con 0x7fe088004800 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a00af880 con 0x7fe088004800 2024-04-25T15:19:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe09000fb40 con 0x7fe088004800 2024-04-25T15:19:21.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.897+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe090013480 con 0x7fe088004800 2024-04-25T15:19:21.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.898+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a006a0e0 con 0x7fe088004800 2024-04-25T15:19:21.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.900+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe0a01e7000 con 0x7fe088004800 2024-04-25T15:19:21.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.966+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fe0a00d2a60 con 0x7fe088004800 2024-04-25T15:19:21.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.968+0000 7fe09d7fa640 1 -- 172.21.15.146:0/3806367574 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fe09001d460 con 0x7fe088004800 2024-04-25T15:19:21.970 INFO:teuthology.orchestra.run.smithi146.stderr:adjusted tunables profile to default 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 msgr2=0x7fe0a00c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe0a00c7d80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe090007840 tx=0x7fe0900018c0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 shutdown_connections 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0a01e6bf0 0x7fe0a00b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0a0151d60 0x7fe0a01e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 --2- 172.21.15.146:0/3806367574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe088004800 0x7fe0a00c7d80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 >> 172.21.15.146:0/3806367574 conn(0x7fe0a00b7490 msgr2=0x7fe0a00b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 shutdown_connections 2024-04-25T15:19:21.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:21.969+0000 7fe0a71c5640 1 -- 172.21.15.146:0/3806367574 wait complete. 2024-04-25T15:19:21.982 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T15:19:21.982 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T15:19:21.983 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:21.983 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T15:19:21.999 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:21.997+0000 7f8fe8123040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:19:22.006 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:22.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T15:19:22.015 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:22.014+0000 7fdf75633040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:19:22.072 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:22.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T15:19:22.137 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:22.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T15:19:22.151 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:22.149+0000 7f8fe8123040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:19:22.174 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:22.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T15:19:22.177 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:22.176+0000 7fdf75633040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:19:22.198 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:22.198 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T15:19:22.207 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:22.205+0000 7f8fe8123040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:19:22.234 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:22.232+0000 7fdf75633040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:19:22.256 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:22.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T15:19:22.320 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:19:22.321 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T15:19:22.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new f99c8f48-3894-494a-9239-7f00941858ab 0 2024-04-25T15:19:22.446 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:22.445+0000 7f8fe8123040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:19:22.491 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:22.489+0000 7fdf75633040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:19:22.492 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:22.491+0000 7f8fe8123040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:19:22.532 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.531+0000 7f4423c61640 1 Processor -- start 2024-04-25T15:19:22.532 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.531+0000 7f4423c61640 1 -- start start 2024-04-25T15:19:22.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f4423c61640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f441c1602e0 0x7f441c0b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f4423c61640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c0cd290 0x7f441c0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f4423c61640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f441c0d2480 con 0x7f441c09b5e0 2024-04-25T15:19:22.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f4423c61640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f441c1606f0 con 0x7f441c1602e0 2024-04-25T15:19:22.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f4423c61640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f441c0d2740 con 0x7f441c0cd290 2024-04-25T15:19:22.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f44219fd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f441c1602e0 0x7f441c0b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f44221fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c0cd290 0x7f441c0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f44211fc640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f441c09b5e0 0x7f441c09b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:50078/0 (socket says 172.21.15.177:50078) 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.532+0000 7f44211fc640 1 -- 172.21.15.177:0/2959029319 learned_addr learned my addr 172.21.15.177:0/2959029319 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44219fd640 1 -- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c0cd290 msgr2=0x7f441c0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44219fd640 1 --2- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c0cd290 0x7f441c0cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44219fd640 1 -- 172.21.15.177:0/2959029319 >> v1:172.21.15.177:6789/0 conn(0x7f441c09b5e0 legacy=0x7f441c09b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44209fb640 1 -- 172.21.15.177:0/2959029319 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 549308692 0 0) 0x7f441c0d2480 con 0x7f441c09b5e0 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44219fd640 1 -- 172.21.15.177:0/2959029319 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441c0d2a20 con 0x7f441c1602e0 2024-04-25T15:19:22.535 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44221fe640 1 --2- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c0cd290 0x7f441c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:22.536 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44219fd640 1 --2- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c1602e0 0x7f441c0b18c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6057e0c0d19ef14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.533+0000 7f44209fb640 1 -- 172.21.15.177:0/2959029319 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c037070 con 0x7f441c1602e0 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f44209fb640 1 -- 172.21.15.177:0/2959029319 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440c032440 con 0x7f441c1602e0 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f44209fb640 1 -- 172.21.15.177:0/2959029319 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c032810 con 0x7f441c1602e0 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 -- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c1602e0 msgr2=0x7f441c0b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 --2- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c1602e0 0x7f441c0b18c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02a180 comp rx=0 tx=0).stop 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 -- 172.21.15.177:0/2959029319 shutdown_connections 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 --2- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c0cd290 0x7f441c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 --2- 172.21.15.177:0/2959029319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c1602e0 0x7f441c0b18c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 -- 172.21.15.177:0/2959029319 >> 172.21.15.177:0/2959029319 conn(0x7f441c0b7490 msgr2=0x7f441c0b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 -- 172.21.15.177:0/2959029319 shutdown_connections 2024-04-25T15:19:22.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.534+0000 7f4423c61640 1 -- 172.21.15.177:0/2959029319 wait complete. 2024-04-25T15:19:22.538 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.536+0000 7f4423c61640 1 Processor -- start 2024-04-25T15:19:22.538 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.536+0000 7f4423c61640 1 -- start start 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.537+0000 7f4423c61640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f441c09b5e0 0x7f441c1f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 0x7f441c15f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 0x7f441c15f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f441c06f7f0 con 0x7f441c09b5e0 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f441c0d2fe0 con 0x7f441c0cd290 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f441c0b1e00 con 0x7f441c1602e0 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44211fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 0x7f441c15f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 0x7f441c15f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44211fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 0x7f441c15f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53548/0 (socket says 172.21.15.177:53548) 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 0x7f441c15f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54380/0 (socket says 172.21.15.177:54380) 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44211fc640 1 -- 172.21.15.177:0/3604137915 learned_addr learned my addr 172.21.15.177:0/3604137915 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44219fd640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f441c09b5e0 0x7f441c1f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 -- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 msgr2=0x7f441c15f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 0x7f441c15f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 -- 172.21.15.177:0/3604137915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f441c09b5e0 msgr2=0x7f441c1f5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f441c09b5e0 0x7f441c1f5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 -- 172.21.15.177:0/3604137915 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441c161260 con 0x7f441c1602e0 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44219fd640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f441c09b5e0 0x7f441c1f5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44211fc640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 0x7f441c15f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44221fe640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 0x7f441c15f600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f441400efc0 tx=0x7f441400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:22.542 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4414018030 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f441400cd30 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441c161540 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4414003bb0 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.538+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f441c0c8f10 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.539+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f441401b070 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.539+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f441401d790 con 0x7f441c1602e0 2024-04-25T15:19:22.543 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:22.539+0000 7fdf75633040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:19:22.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.540+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f441c06a0e0 con 0x7f441c1602e0 2024-04-25T15:19:22.544 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.542+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4414010040 con 0x7f441c1602e0 2024-04-25T15:19:22.617 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.615+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f99c8f48-3894-494a-9239-7f00941858ab", "id": 0} v 0) -- 0x7f441c15d960 con 0x7f441c1602e0 2024-04-25T15:19:22.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.618+0000 7f44127fc640 1 -- 172.21.15.177:0/3604137915 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f99c8f48-3894-494a-9239-7f00941858ab", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f441401d9e0 con 0x7f441c1602e0 2024-04-25T15:19:22.620 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 msgr2=0x7f441c15f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 0x7f441c15f600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f441400efc0 tx=0x7f441400c490 comp rx=0 tx=0).stop 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 shutdown_connections 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f441c1602e0 0x7f441c15f600 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f441c0cd290 0x7f441c15f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 --2- 172.21.15.177:0/3604137915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f441c09b5e0 0x7f441c1f5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 >> 172.21.15.177:0/3604137915 conn(0x7f441c0b7490 msgr2=0x7f441c06edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 shutdown_connections 2024-04-25T15:19:22.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.619+0000 7f4423c61640 1 -- 172.21.15.177:0/3604137915 wait complete. 2024-04-25T15:19:22.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 13de8f11-bab2-4388-a48e-d1ef86f9ac75 1 2024-04-25T15:19:22.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.737+0000 7fab21b6c640 1 Processor -- start 2024-04-25T15:19:22.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.737+0000 7fab21b6c640 1 -- start start 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.738+0000 7fab21b6c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fab1c15ced0 0x7fab1c09bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.738+0000 7fab21b6c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c0d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.738+0000 7fab21b6c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c15d2e0 con 0x7fab1c06def0 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.738+0000 7fab21b6c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c0d2fe0 con 0x7fab1c15ced0 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.738+0000 7fab21b6c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c161850 con 0x7fab1c0d2690 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c0d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab1b7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fab1c15ced0 0x7fab1c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c0d2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54386/0 (socket says 172.21.15.177:54386) 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 -- 172.21.15.177:0/864943820 learned_addr learned my addr 172.21.15.177:0/864943820 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 -- 172.21.15.177:0/864943820 >> v2:172.21.15.177:3300/0 conn(0x7fab1c15ced0 msgr2=0x7fab1c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 --2- 172.21.15.177:0/864943820 >> v2:172.21.15.177:3300/0 conn(0x7fab1c15ced0 0x7fab1c09bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 -- 172.21.15.177:0/864943820 >> v1:172.21.15.146:6789/0 conn(0x7fab1c06def0 legacy=0x7fab1c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab13fff640 1 -- 172.21.15.177:0/864943820 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c06f360 con 0x7fab1c0d2690 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.739+0000 7fab1b7fe640 1 --2- 172.21.15.177:0/864943820 >> v2:172.21.15.177:3300/0 conn(0x7fab1c15ced0 0x7fab1c09bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.740+0000 7fab13fff640 1 --2- 172.21.15.177:0/864943820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c0d2aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab0400bb10 tx=0x7fab0402c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f7182242c69b48d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.740+0000 7fab1affd640 1 -- 172.21.15.177:0/864943820 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04036020 con 0x7fab1c0d2690 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.740+0000 7fab1affd640 1 -- 172.21.15.177:0/864943820 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab04032820 con 0x7fab1c0d2690 2024-04-25T15:19:22.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.740+0000 7fab1affd640 1 -- 172.21.15.177:0/864943820 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04032bf0 con 0x7fab1c0d2690 2024-04-25T15:19:22.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.740+0000 7fab21b6c640 1 -- 172.21.15.177:0/864943820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 msgr2=0x7fab1c0d2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.740+0000 7fab21b6c640 1 --2- 172.21.15.177:0/864943820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c0d2aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab0400bb10 tx=0x7fab0402c920 comp rx=0 tx=0).stop 2024-04-25T15:19:22.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.741+0000 7fab21b6c640 1 -- 172.21.15.177:0/864943820 shutdown_connections 2024-04-25T15:19:22.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.741+0000 7fab21b6c640 1 --2- 172.21.15.177:0/864943820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c0d2aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.741+0000 7fab21b6c640 1 --2- 172.21.15.177:0/864943820 >> v2:172.21.15.177:3300/0 conn(0x7fab1c15ced0 0x7fab1c09bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.741+0000 7fab21b6c640 1 -- 172.21.15.177:0/864943820 >> 172.21.15.177:0/864943820 conn(0x7fab1c0b7430 msgr2=0x7fab1c0b7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:22.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.741+0000 7fab21b6c640 1 -- 172.21.15.177:0/864943820 shutdown_connections 2024-04-25T15:19:22.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.741+0000 7fab21b6c640 1 -- 172.21.15.177:0/864943820 wait complete. 2024-04-25T15:19:22.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.742+0000 7fab21b6c640 1 Processor -- start 2024-04-25T15:19:22.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.742+0000 7fab21b6c640 1 -- start start 2024-04-25T15:19:22.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 0x7fab1c1fa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c14c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab1c15ced0 0x7fab1c0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c09b8b0 con 0x7fab1c15ced0 2024-04-25T15:19:22.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0d0470 con 0x7fab1c06def0 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c1613c0 con 0x7fab1c0d2690 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 0x7fab1c1fa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab13fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c14c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab13fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c14c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54392/0 (socket says 172.21.15.177:54392) 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 0x7fab1c1fa4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53568/0 (socket says 172.21.15.177:53568) 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab13fff640 1 -- 172.21.15.177:0/506053681 learned_addr learned my addr 172.21.15.177:0/506053681 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1bfff640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab1c15ced0 0x7fab1c0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 -- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 msgr2=0x7fab1c14c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c14c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 -- 172.21.15.177:0/506053681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab1c15ced0 msgr2=0x7fab1c0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab1c15ced0 0x7fab1c0b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 -- 172.21.15.177:0/506053681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c149630 con 0x7fab1c06def0 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab13fff640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c14c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:22.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab1b7fe640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 0x7fab1c1fa4a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00e8a0 tx=0x7fab0c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c017070 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab0c00c940 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c15d750 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00cd30 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.744+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c1f9ca0 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.745+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fab0c01a070 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.745+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fab0c013470 con 0x7fab1c06def0 2024-04-25T15:19:22.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.746+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1c06a0e0 con 0x7fab1c06def0 2024-04-25T15:19:22.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.748+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fab0c009300 con 0x7fab1c06def0 2024-04-25T15:19:22.821 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.820+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "13de8f11-bab2-4388-a48e-d1ef86f9ac75", "id": 1} v 0) -- 0x7fab1c159e50 con 0x7fab1c06def0 2024-04-25T15:19:22.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.825+0000 7fab18ff9640 1 -- 172.21.15.177:0/506053681 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "13de8f11-bab2-4388-a48e-d1ef86f9ac75", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fab0c006020 con 0x7fab1c06def0 2024-04-25T15:19:22.827 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 msgr2=0x7fab1c1fa4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 0x7fab1c1fa4a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00e8a0 tx=0x7fab0c00ed60 comp rx=0 tx=0).stop 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 shutdown_connections 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab1c15ced0 0x7fab1c0b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab1c0d2690 0x7fab1c14c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 --2- 172.21.15.177:0/506053681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab1c06def0 0x7fab1c1fa4a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 >> 172.21.15.177:0/506053681 conn(0x7fab1c0b7430 msgr2=0x7fab1c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:22.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 shutdown_connections 2024-04-25T15:19:22.829 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.826+0000 7fab21b6c640 1 -- 172.21.15.177:0/506053681 wait complete. 2024-04-25T15:19:22.839 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 86c182b9-706e-4b35-807b-94719fd66e36 2 2024-04-25T15:19:22.949 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.948+0000 7f03c8334640 1 Processor -- start 2024-04-25T15:19:22.949 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.948+0000 7f03c8334640 1 -- start start 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c8334640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03c01602e0 0x7f03c00b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c8334640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 0x7f03c009b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c8334640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03c009bf30 con 0x7f03c00cf130 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c8334640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f03c01606f0 con 0x7f03c01602e0 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c8334640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f03c00d2480 con 0x7f03c009b5e0 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c58cf640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 0x7f03c009b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.952 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c58cf640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 0x7f03c009b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54398/0 (socket says 172.21.15.177:54398) 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c58cf640 1 -- 172.21.15.177:0/1890374187 learned_addr learned my addr 172.21.15.177:0/1890374187 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c60d0640 1 --2- 172.21.15.177:0/1890374187 >> v2:172.21.15.146:3300/0 conn(0x7f03c01602e0 0x7f03c00b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c58cf640 1 -- 172.21.15.177:0/1890374187 >> v2:172.21.15.146:3300/0 conn(0x7f03c01602e0 msgr2=0x7f03c00b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c58cf640 1 --2- 172.21.15.177:0/1890374187 >> v2:172.21.15.146:3300/0 conn(0x7f03c01602e0 0x7f03c00b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.949+0000 7f03c58cf640 1 -- 172.21.15.177:0/1890374187 >> v1:172.21.15.146:6789/0 conn(0x7f03c00cf130 legacy=0x7f03c00cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.950+0000 7f03c58cf640 1 -- 172.21.15.177:0/1890374187 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c00d2740 con 0x7f03c009b5e0 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.950+0000 7f03c58cf640 1 --2- 172.21.15.177:0/1890374187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 0x7f03c009b9f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009870 tx=0x7f03b002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad90eddd55c508d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.950+0000 7f03c50ce640 1 -- 172.21.15.177:0/1890374187 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b0037070 con 0x7f03c009b5e0 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.950+0000 7f03c50ce640 1 -- 172.21.15.177:0/1890374187 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03b0032440 con 0x7f03c009b5e0 2024-04-25T15:19:22.953 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.950+0000 7f03c50ce640 1 -- 172.21.15.177:0/1890374187 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b0032810 con 0x7f03c009b5e0 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 -- 172.21.15.177:0/1890374187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 msgr2=0x7f03c009b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 --2- 172.21.15.177:0/1890374187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 0x7f03c009b9f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009870 tx=0x7f03b002a270 comp rx=0 tx=0).stop 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 -- 172.21.15.177:0/1890374187 shutdown_connections 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 --2- 172.21.15.177:0/1890374187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c009b5e0 0x7f03c009b9f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 --2- 172.21.15.177:0/1890374187 >> v2:172.21.15.146:3300/0 conn(0x7f03c01602e0 0x7f03c00b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 -- 172.21.15.177:0/1890374187 >> 172.21.15.177:0/1890374187 conn(0x7f03c00b7490 msgr2=0x7f03c00b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 -- 172.21.15.177:0/1890374187 shutdown_connections 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.951+0000 7f03c8334640 1 -- 172.21.15.177:0/1890374187 wait complete. 2024-04-25T15:19:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.953+0000 7f03c8334640 1 Processor -- start 2024-04-25T15:19:22.956 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.953+0000 7f03c8334640 1 -- start start 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c8334640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03c009b5e0 0x7f03c00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c8334640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 0x7f03c01591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c8334640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c01602e0 0x7f03c006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c8334640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03c00d2fe0 con 0x7f03c01602e0 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c8334640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03c00d2d00 con 0x7f03c009b5e0 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c8334640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f03c006f7f0 con 0x7f03c00cf130 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c58cf640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 0x7f03c01591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.954+0000 7f03c60d0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03c009b5e0 0x7f03c00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c58cf640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 0x7f03c01591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54402/0 (socket says 172.21.15.177:54402) 2024-04-25T15:19:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c58cf640 1 -- 172.21.15.177:0/2418277802 learned_addr learned my addr 172.21.15.177:0/2418277802 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c68d1640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c01602e0 0x7f03c006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c60d0640 1 -- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 msgr2=0x7f03c01591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c60d0640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 0x7f03c01591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c60d0640 1 -- 172.21.15.177:0/2418277802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c01602e0 msgr2=0x7f03c006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c60d0640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c01602e0 0x7f03c006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c60d0640 1 -- 172.21.15.177:0/2418277802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c00c72f0 con 0x7f03c009b5e0 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c58cf640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 0x7f03c01591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c60d0640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03c009b5e0 0x7f03c00cb560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03c0163cd0 tx=0x7f03bc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc017070 con 0x7f03c009b5e0 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03bc00c730 con 0x7f03c009b5e0 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c00c75d0 con 0x7f03c009b5e0 2024-04-25T15:19:22.960 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00cb20 con 0x7f03c009b5e0 2024-04-25T15:19:22.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c00c7b90 con 0x7f03c009b5e0 2024-04-25T15:19:22.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f03bc01a070 con 0x7f03c009b5e0 2024-04-25T15:19:22.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.955+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f03bc01c870 con 0x7f03c009b5e0 2024-04-25T15:19:22.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.956+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c0068c30 con 0x7f03c009b5e0 2024-04-25T15:19:22.961 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:22.958+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f03c0068c30 con 0x7f03c009b5e0 2024-04-25T15:19:23.031 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.030+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "86c182b9-706e-4b35-807b-94719fd66e36", "id": 2} v 0) -- 0x7f03c006a0e0 con 0x7f03c009b5e0 2024-04-25T15:19:23.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.035+0000 7f03b6ffd640 1 -- 172.21.15.177:0/2418277802 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "86c182b9-706e-4b35-807b-94719fd66e36", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f03bc01c450 con 0x7f03c009b5e0 2024-04-25T15:19:23.037 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-04-25T15:19:23.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03c009b5e0 msgr2=0x7f03c00cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03c009b5e0 0x7f03c00cb560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03c0163cd0 tx=0x7f03bc00ecd0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 shutdown_connections 2024-04-25T15:19:23.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c01602e0 0x7f03c006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03c00cf130 0x7f03c01591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 --2- 172.21.15.177:0/2418277802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03c009b5e0 0x7f03c00cb560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 >> 172.21.15.177:0/2418277802 conn(0x7f03c00b7490 msgr2=0x7f03c015f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 shutdown_connections 2024-04-25T15:19:23.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.036+0000 7f03c8334640 1 -- 172.21.15.177:0/2418277802 wait complete. 2024-04-25T15:19:23.049 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new abe109f1-9591-4742-9d25-d7595f02c506 3 2024-04-25T15:19:23.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.156+0000 7f6147fb7640 1 Processor -- start 2024-04-25T15:19:23.158 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.157+0000 7f6147fb7640 1 -- start start 2024-04-25T15:19:23.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6147fb7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f614009b5e0 0x7f614009b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.160 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6147fb7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61400cd290 0x7f61400cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6147fb7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f614009bf30 con 0x7f61401602e0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6147fb7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f61401606f0 con 0x7f61400cd290 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6147fb7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f61400d2480 con 0x7f614009b5e0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f614009b5e0 0x7f614009b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f614009b5e0 0x7f614009b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53590/0 (socket says 172.21.15.177:53590) 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 -- 172.21.15.177:0/1743250373 learned_addr learned my addr 172.21.15.177:0/1743250373 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6146554640 1 --2- 172.21.15.177:0/1743250373 >> v2:172.21.15.146:3300/0 conn(0x7f61400cd290 0x7f61400cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 -- 172.21.15.177:0/1743250373 >> v2:172.21.15.146:3300/0 conn(0x7f61400cd290 msgr2=0x7f61400cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 --2- 172.21.15.177:0/1743250373 >> v2:172.21.15.146:3300/0 conn(0x7f61400cd290 0x7f61400cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 -- 172.21.15.177:0/1743250373 >> v1:172.21.15.146:6789/0 conn(0x7f61401602e0 legacy=0x7f61400b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.158+0000 7f6145552640 1 -- 172.21.15.177:0/1743250373 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61400d2740 con 0x7f614009b5e0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.159+0000 7f6145552640 1 --2- 172.21.15.177:0/1743250373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f614009b5e0 0x7f614009b9f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f614009c440 tx=0x7f613c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=482ab3f0366d23ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.159+0000 7f6144d51640 1 -- 172.21.15.177:0/1743250373 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c037070 con 0x7f614009b5e0 2024-04-25T15:19:23.161 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.159+0000 7f6144d51640 1 -- 172.21.15.177:0/1743250373 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613c032440 con 0x7f614009b5e0 2024-04-25T15:19:23.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.159+0000 7f6144d51640 1 -- 172.21.15.177:0/1743250373 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c032810 con 0x7f614009b5e0 2024-04-25T15:19:23.162 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 -- 172.21.15.177:0/1743250373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f614009b5e0 msgr2=0x7f614009b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1743250373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f614009b5e0 0x7f614009b9f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f614009c440 tx=0x7f613c02a270 comp rx=0 tx=0).stop 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 -- 172.21.15.177:0/1743250373 shutdown_connections 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1743250373 >> v2:172.21.15.146:3300/0 conn(0x7f61400cd290 0x7f61400cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1743250373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f614009b5e0 0x7f614009b9f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 -- 172.21.15.177:0/1743250373 >> 172.21.15.177:0/1743250373 conn(0x7f61400b7490 msgr2=0x7f61400b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 -- 172.21.15.177:0/1743250373 shutdown_connections 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.160+0000 7f6147fb7640 1 -- 172.21.15.177:0/1743250373 wait complete. 2024-04-25T15:19:23.163 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.162+0000 7f6147fb7640 1 Processor -- start 2024-04-25T15:19:23.164 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.162+0000 7f6147fb7640 1 -- start start 2024-04-25T15:19:23.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6147fb7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f614009b5e0 0x7f6140159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6147fb7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 0x7f61400c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6147fb7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61401602e0 0x7f614006e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6147fb7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61400d2fe0 con 0x7f614009b5e0 2024-04-25T15:19:23.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6147fb7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61400d2d00 con 0x7f61400cd290 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6147fb7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f614006f7f0 con 0x7f61401602e0 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6145552640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 0x7f61400c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6146554640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61401602e0 0x7f614006e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6145552640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 0x7f61400c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53602/0 (socket says 172.21.15.177:53602) 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6145552640 1 -- 172.21.15.177:0/1031874362 learned_addr learned my addr 172.21.15.177:0/1031874362 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.163+0000 7f6145d53640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f614009b5e0 0x7f6140159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6145552640 1 -- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61401602e0 msgr2=0x7f614006e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6145552640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61401602e0 0x7f614006e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6145552640 1 -- 172.21.15.177:0/1031874362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f614009b5e0 msgr2=0x7f6140159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6145552640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f614009b5e0 0x7f6140159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6145552640 1 -- 172.21.15.177:0/1031874362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61400c8a60 con 0x7f61400cd290 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6146554640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61401602e0 0x7f614006e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6145552640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 0x7f61400c6ba0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f613c000c00 tx=0x7f613c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c037040 con 0x7f61400cd290 2024-04-25T15:19:23.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613c042070 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61400c8d40 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613c030860 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61400c9300 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f613c030a00 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.164+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f613c03b430 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.165+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614006a0e0 con 0x7f61400cd290 2024-04-25T15:19:23.168 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.167+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f613c047460 con 0x7f61400cd290 2024-04-25T15:19:23.196 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.194+0000 7f8fe8123040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:19:23.243 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.241+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "abe109f1-9591-4742-9d25-d7595f02c506", "id": 3} v 0) -- 0x7f61400717d0 con 0x7f61400cd290 2024-04-25T15:19:23.248 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.247+0000 7f61367fc640 1 -- 172.21.15.177:0/1031874362 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "abe109f1-9591-4742-9d25-d7595f02c506", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f613c047700 con 0x7f61400cd290 2024-04-25T15:19:23.249 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 msgr2=0x7f61400c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 0x7f61400c6ba0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f613c000c00 tx=0x7f613c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 shutdown_connections 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61401602e0 0x7f614006e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61400cd290 0x7f61400c6ba0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 --2- 172.21.15.177:0/1031874362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f614009b5e0 0x7f6140159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 >> 172.21.15.177:0/1031874362 conn(0x7f61400b7490 msgr2=0x7f61400aff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 shutdown_connections 2024-04-25T15:19:23.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.248+0000 7f6147fb7640 1 -- 172.21.15.177:0/1031874362 wait complete. 2024-04-25T15:19:23.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 5bff8dde-d43c-4e63-9e32-d38b5cd88226 4 2024-04-25T15:19:23.279 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.278+0000 7fdf75633040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:19:23.283 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.281+0000 7f8fe8123040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:19:23.325 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.323+0000 7f8fe8123040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:19:23.368 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.367+0000 7fe4c5005640 1 Processor -- start 2024-04-25T15:19:23.368 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.367+0000 7fe4c5005640 1 -- start start 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4c5005640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4c009b5e0 0x7fe4c009b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4c5005640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c009bf30 con 0x7fe4c00cd290 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4c5005640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c00d2480 con 0x7fe4c01602e0 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4c5005640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c01606f0 con 0x7fe4c009b5e0 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4bed9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe4c01602e0 0x7fe4c00b18c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:50088/0 (socket says 172.21.15.177:50088) 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4bed9d640 1 -- 172.21.15.177:0/3577544722 learned_addr learned my addr 172.21.15.177:0/3577544722 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4be59c640 1 --2- 172.21.15.177:0/3577544722 >> v2:172.21.15.146:3300/0 conn(0x7fe4c009b5e0 0x7fe4c009b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.368+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751120977 0 0) 0x7fe4c00d2480 con 0x7fe4c01602e0 2024-04-25T15:19:23.371 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4a8003520 con 0x7fe4c01602e0 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752658342 0 0) 0x7fe4b8001490 con 0x7fe4c00cd290 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4c00d2480 con 0x7fe4c00cd290 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4be59c640 1 -- 172.21.15.177:0/3577544722 >> v1:172.21.15.177:6789/0 conn(0x7fe4c01602e0 legacy=0x7fe4c00b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4be59c640 1 -- 172.21.15.177:0/3577544722 >> v1:172.21.15.146:6789/0 conn(0x7fe4c00cd290 legacy=0x7fe4c00cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4be59c640 1 -- 172.21.15.177:0/3577544722 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c00d2740 con 0x7fe4c009b5e0 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.369+0000 7fe4be59c640 1 --2- 172.21.15.177:0/3577544722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c009b5e0 0x7fe4c009b9f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40097a0 tx=0x7fe4b402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb623b2e4eb168e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4037070 con 0x7fe4c009b5e0 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b4032440 con 0x7fe4c009b5e0 2024-04-25T15:19:23.372 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4bdd9b640 1 -- 172.21.15.177:0/3577544722 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b40327f0 con 0x7fe4c009b5e0 2024-04-25T15:19:23.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4c5005640 1 -- 172.21.15.177:0/3577544722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c009b5e0 msgr2=0x7fe4c009b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4c5005640 1 --2- 172.21.15.177:0/3577544722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c009b5e0 0x7fe4c009b9f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40097a0 tx=0x7fe4b402a230 comp rx=0 tx=0).stop 2024-04-25T15:19:23.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4c5005640 1 -- 172.21.15.177:0/3577544722 shutdown_connections 2024-04-25T15:19:23.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4c5005640 1 --2- 172.21.15.177:0/3577544722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c009b5e0 0x7fe4c009b9f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4c5005640 1 -- 172.21.15.177:0/3577544722 >> 172.21.15.177:0/3577544722 conn(0x7fe4c00b7490 msgr2=0x7fe4c00b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.370+0000 7fe4c5005640 1 -- 172.21.15.177:0/3577544722 shutdown_connections 2024-04-25T15:19:23.374 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.371+0000 7fe4c5005640 1 -- 172.21.15.177:0/3577544722 wait complete. 2024-04-25T15:19:23.374 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.372+0000 7fdf75633040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:19:23.374 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.373+0000 7fe4c5005640 1 Processor -- start 2024-04-25T15:19:23.374 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.373+0000 7fe4c5005640 1 -- start start 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4c5005640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 0x7fe4c015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4c5005640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 0x7fe4c00b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4c5005640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c01602e0 0x7fe4c00c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4c5005640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c00d2fe0 con 0x7fe4c01602e0 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4c5005640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c00d2d00 con 0x7fe4c009b5e0 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4c5005640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c006f7f0 con 0x7fe4c00cd290 2024-04-25T15:19:23.377 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4bed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 0x7fe4c015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 0x7fe4c00b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 0x7fe4c00b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54416/0 (socket says 172.21.15.177:54416) 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 -- 172.21.15.177:0/620854705 learned_addr learned my addr 172.21.15.177:0/620854705 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4bed9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 0x7fe4c015baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53616/0 (socket says 172.21.15.177:53616) 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4bf59e640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c01602e0 0x7fe4c00c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 -- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 msgr2=0x7fe4c015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 0x7fe4c015baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 -- 172.21.15.177:0/620854705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c01602e0 msgr2=0x7fe4c00c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c01602e0 0x7fe4c00c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.378 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4be59c640 1 -- 172.21.15.177:0/620854705 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c00cad20 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4bed9d640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 0x7fe4c015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.374+0000 7fe4bf59e640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c01602e0 0x7fe4c00c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.375+0000 7fe4be59c640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 0x7fe4c00b4bb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000c00 tx=0x7fe4b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.375+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4041020 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.375+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b403d620 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.375+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4037040 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.375+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c00c64c0 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.375+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c015c030 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.376+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe4b4044020 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.376+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fe4b403dda0 con 0x7fe4c00cd290 2024-04-25T15:19:23.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.376+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c006a0e0 con 0x7fe4c00cd290 2024-04-25T15:19:23.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.378+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe4b404b030 con 0x7fe4c00cd290 2024-04-25T15:19:23.418 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.417+0000 7fdf75633040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:19:23.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.450+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5bff8dde-d43c-4e63-9e32-d38b5cd88226", "id": 4} v 0) -- 0x7fe4c006da20 con 0x7fe4c00cd290 2024-04-25T15:19:23.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.457+0000 7fe4a77fe640 1 -- 172.21.15.177:0/620854705 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5bff8dde-d43c-4e63-9e32-d38b5cd88226", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fe4b4046cc0 con 0x7fe4c00cd290 2024-04-25T15:19:23.458 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-04-25T15:19:23.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 msgr2=0x7fe4c00b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 0x7fe4c00b4bb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000c00 tx=0x7fe4b40046c0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 shutdown_connections 2024-04-25T15:19:23.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c01602e0 0x7fe4c00c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4c00cd290 0x7fe4c00b4bb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 --2- 172.21.15.177:0/620854705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4c009b5e0 0x7fe4c015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 >> 172.21.15.177:0/620854705 conn(0x7fe4c00b7490 msgr2=0x7fe4c006e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 shutdown_connections 2024-04-25T15:19:23.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.458+0000 7fe4c5005640 1 -- 172.21.15.177:0/620854705 wait complete. 2024-04-25T15:19:23.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new e00f58da-2f23-448c-ac44-7673c0ab01e7 5 2024-04-25T15:19:23.491 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.490+0000 7f8fe8123040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:19:23.546 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.544+0000 7f8fe8123040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:19:23.593 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.592+0000 7fdf75633040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:19:23.598 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.596+0000 7f8fe8123040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:19:23.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.616+0000 7ffaf6a53640 1 Processor -- start 2024-04-25T15:19:23.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.616+0000 7ffaf6a53640 1 -- start start 2024-04-25T15:19:23.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf6a53640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffaf015a0e0 0x7ffaf00b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf6a53640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffaf00d2240 0x7ffaf006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf6a53640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaf015da20 con 0x7ffaf015ca20 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf6a53640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffaf00b79b0 con 0x7ffaf015a0e0 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf6a53640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ffaf006d010 con 0x7ffaf00d2240 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffaf00d2240 0x7ffaf006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffaf00d2240 0x7ffaf006d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53628/0 (socket says 172.21.15.177:53628) 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 -- 172.21.15.177:0/3627059301 learned_addr learned my addr 172.21.15.177:0/3627059301 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaeffff640 1 --2- 172.21.15.177:0/3627059301 >> v2:172.21.15.146:3300/0 conn(0x7ffaf015a0e0 0x7ffaf00b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 -- 172.21.15.177:0/3627059301 >> v2:172.21.15.146:3300/0 conn(0x7ffaf015a0e0 msgr2=0x7ffaf00b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 --2- 172.21.15.177:0/3627059301 >> v2:172.21.15.146:3300/0 conn(0x7ffaf015a0e0 0x7ffaf00b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 -- 172.21.15.177:0/3627059301 >> v1:172.21.15.146:6789/0 conn(0x7ffaf015ca20 legacy=0x7ffaf00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.618+0000 7ffaf4ff0640 1 -- 172.21.15.177:0/3627059301 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf015d180 con 0x7ffaf00d2240 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.619+0000 7ffaf4ff0640 1 --2- 172.21.15.177:0/3627059301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf00d2240 0x7ffaf006d4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00746e0 tx=0x7ffae802c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9876571b20caa862 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.619+0000 7ffaeeffd640 1 -- 172.21.15.177:0/3627059301 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae80335b0 con 0x7ffaf00d2240 2024-04-25T15:19:23.621 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.619+0000 7ffaeeffd640 1 -- 172.21.15.177:0/3627059301 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae8039070 con 0x7ffaf00d2240 2024-04-25T15:19:23.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.619+0000 7ffaeeffd640 1 -- 172.21.15.177:0/3627059301 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae8033970 con 0x7ffaf00d2240 2024-04-25T15:19:23.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 -- 172.21.15.177:0/3627059301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf00d2240 msgr2=0x7ffaf006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.622 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/3627059301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf00d2240 0x7ffaf006d4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00746e0 tx=0x7ffae802c940 comp rx=0 tx=0).stop 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 -- 172.21.15.177:0/3627059301 shutdown_connections 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/3627059301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf00d2240 0x7ffaf006d4a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/3627059301 >> v2:172.21.15.146:3300/0 conn(0x7ffaf015a0e0 0x7ffaf00b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 -- 172.21.15.177:0/3627059301 >> 172.21.15.177:0/3627059301 conn(0x7ffaf0073e50 msgr2=0x7ffaf009b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 -- 172.21.15.177:0/3627059301 shutdown_connections 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.620+0000 7ffaf6a53640 1 -- 172.21.15.177:0/3627059301 wait complete. 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.622+0000 7ffaf6a53640 1 Processor -- start 2024-04-25T15:19:23.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.622+0000 7ffaf6a53640 1 -- start start 2024-04-25T15:19:23.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf6a53640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 0x7ffaf0150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf6a53640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaf015ca20 0x7ffaf00ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf6a53640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaf01f1800 0x7ffaf01f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf6a53640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf015ce30 con 0x7ffaf01f1800 2024-04-25T15:19:23.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf6a53640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf015d740 con 0x7ffaf015a0e0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf6a53640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffaf006c550 con 0x7ffaf015ca20 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaeffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 0x7ffaf0150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaef7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaf015ca20 0x7ffaf00ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaeffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 0x7ffaf0150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53644/0 (socket says 172.21.15.177:53644) 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaeffff640 1 -- 172.21.15.177:0/995057481 learned_addr learned my addr 172.21.15.177:0/995057481 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.623+0000 7ffaf4ff0640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaf01f1800 0x7ffaf01f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaeffff640 1 -- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaf015ca20 msgr2=0x7ffaf00ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaeffff640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaf015ca20 0x7ffaf00ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaeffff640 1 -- 172.21.15.177:0/995057481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaf01f1800 msgr2=0x7ffaf01f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaeffff640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaf01f1800 0x7ffaf01f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaeffff640 1 -- 172.21.15.177:0/995057481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf00cf9d0 con 0x7ffaf015a0e0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaef7fe640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaf015ca20 0x7ffaf00ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaeffff640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 0x7ffaf0150b00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffae400e8a0 tx=0x7ffae400eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4017070 con 0x7ffaf015a0e0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae400c940 con 0x7ffaf015a0e0 2024-04-25T15:19:23.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf00cfcb0 con 0x7ffaf015a0e0 2024-04-25T15:19:23.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae400cd30 con 0x7ffaf015a0e0 2024-04-25T15:19:23.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf00afca0 con 0x7ffaf015a0e0 2024-04-25T15:19:23.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffae401a070 con 0x7ffaf015a0e0 2024-04-25T15:19:23.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.624+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7ffae401c9d0 con 0x7ffaf015a0e0 2024-04-25T15:19:23.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.625+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaf006a0e0 con 0x7ffaf015a0e0 2024-04-25T15:19:23.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.627+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ffae401c450 con 0x7ffaf015a0e0 2024-04-25T15:19:23.659 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.658+0000 7fdf75633040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:19:23.700 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.699+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e00f58da-2f23-448c-ac44-7673c0ab01e7", "id": 5} v 0) -- 0x7ffaf015c350 con 0x7ffaf015a0e0 2024-04-25T15:19:23.706 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.704+0000 7ffaecff9640 1 -- 172.21.15.177:0/995057481 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e00f58da-2f23-448c-ac44-7673c0ab01e7", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7ffae401c630 con 0x7ffaf015a0e0 2024-04-25T15:19:23.706 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-04-25T15:19:23.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 msgr2=0x7ffaf0150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 0x7ffaf0150b00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffae400e8a0 tx=0x7ffae400eca0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 shutdown_connections 2024-04-25T15:19:23.707 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaf01f1800 0x7ffaf01f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.708 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaf015ca20 0x7ffaf00ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.708 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 --2- 172.21.15.177:0/995057481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf015a0e0 0x7ffaf0150b00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.708 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.705+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 >> 172.21.15.177:0/995057481 conn(0x7ffaf0073e50 msgr2=0x7ffaf015b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.708 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.706+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 shutdown_connections 2024-04-25T15:19:23.708 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.706+0000 7ffaf6a53640 1 -- 172.21.15.177:0/995057481 wait complete. 2024-04-25T15:19:23.711 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:23.709+0000 7f8fe8123040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:19:23.714 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.712+0000 7fdf75633040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:19:23.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 2348a56d-1908-4fae-9b45-53aaf09fc42a 6 2024-04-25T15:19:23.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.824+0000 7fa779d7a640 1 Processor -- start 2024-04-25T15:19:23.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.824+0000 7fa779d7a640 1 -- start start 2024-04-25T15:19:23.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa779d7a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa7740cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa779d7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa77408d590 con 0x7fa7740b4ca0 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa779d7a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7740d27d0 con 0x7fa77408dd50 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa779d7a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa7740b9340 con 0x7fa7740b8f30 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa773fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa7740cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa7737fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fa77408dd50 0x7fa7740d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:50098/0 (socket says 172.21.15.177:50098) 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa7737fe640 1 -- 172.21.15.177:0/3685080437 learned_addr learned my addr 172.21.15.177:0/3685080437 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa772ffd640 1 -- 172.21.15.177:0/3685080437 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248860601 0 0) 0x7fa7740d27d0 con 0x7fa77408dd50 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa772ffd640 1 -- 172.21.15.177:0/3685080437 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa760003520 con 0x7fa77408dd50 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa773fff640 1 -- 172.21.15.177:0/3685080437 >> v1:172.21.15.177:6789/0 conn(0x7fa77408dd50 legacy=0x7fa7740d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa773fff640 1 -- 172.21.15.177:0/3685080437 >> v1:172.21.15.146:6789/0 conn(0x7fa7740b4ca0 legacy=0x7fa7740ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.826+0000 7fa773fff640 1 -- 172.21.15.177:0/3685080437 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7740b48b0 con 0x7fa7740b8f30 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.827+0000 7fa773fff640 1 --2- 172.21.15.177:0/3685080437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa7740cee00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009dc0 tx=0x7fa76c02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5495ef972d2ee5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.827+0000 7fa772ffd640 1 -- 172.21.15.177:0/3685080437 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c007740 con 0x7fa7740b8f30 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.827+0000 7fa772ffd640 1 -- 172.21.15.177:0/3685080437 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa76c0078e0 con 0x7fa7740b8f30 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.827+0000 7fa772ffd640 1 -- 172.21.15.177:0/3685080437 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c007cb0 con 0x7fa7740b8f30 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 -- 172.21.15.177:0/3685080437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 msgr2=0x7fa7740cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 --2- 172.21.15.177:0/3685080437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa7740cee00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009dc0 tx=0x7fa76c02a6d0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 -- 172.21.15.177:0/3685080437 shutdown_connections 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 --2- 172.21.15.177:0/3685080437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa7740cee00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 -- 172.21.15.177:0/3685080437 >> 172.21.15.177:0/3685080437 conn(0x7fa7740b5430 msgr2=0x7fa7740b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 -- 172.21.15.177:0/3685080437 shutdown_connections 2024-04-25T15:19:23.830 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.828+0000 7fa779d7a640 1 -- 172.21.15.177:0/3685080437 wait complete. 2024-04-25T15:19:23.831 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.830+0000 7fa779d7a640 1 Processor -- start 2024-04-25T15:19:23.831 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.830+0000 7fa779d7a640 1 -- start start 2024-04-25T15:19:23.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa779d7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7740b4ca0 0x7fa7741d6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa779d7a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa77409f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa779d7a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7741e89a0 0x7fa7741e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:23.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa779d7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7740d2340 con 0x7fa7740b4ca0 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa779d7a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa77408d8c0 con 0x7fa7741e89a0 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa779d7a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa7740d24c0 con 0x7fa7740b8f30 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa77409f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa773fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7741e89a0 0x7fa7741e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa77409f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54432/0 (socket says 172.21.15.177:54432) 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa76bfff640 1 -- 172.21.15.177:0/2868016673 learned_addr learned my addr 172.21.15.177:0/2868016673 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.831+0000 7fa7737fe640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7740b4ca0 0x7fa7741d6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa76bfff640 1 -- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7741e89a0 msgr2=0x7fa7741e8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa76bfff640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7741e89a0 0x7fa7741e8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa76bfff640 1 -- 172.21.15.177:0/2868016673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7740b4ca0 msgr2=0x7fa7741d6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa76bfff640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7740b4ca0 0x7fa7741d6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa76bfff640 1 -- 172.21.15.177:0/2868016673 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7741e92f0 con 0x7fa7740b8f30 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa76bfff640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa77409f250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00c340 tx=0x7fa75c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:23.835 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c017070 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75c0052d0 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77409fcb0 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c003850 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7740a0270 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa75c01a070 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.832+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fa75c011c20 con 0x7fa7740b8f30 2024-04-25T15:19:23.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.833+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa77406a0e0 con 0x7fa7740b8f30 2024-04-25T15:19:23.837 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.836+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa75c021020 con 0x7fa7740b8f30 2024-04-25T15:19:23.837 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:23.836+0000 7fdf75633040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:19:23.912 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.910+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2348a56d-1908-4fae-9b45-53aaf09fc42a", "id": 6} v 0) -- 0x7fa7740c6de0 con 0x7fa7740b8f30 2024-04-25T15:19:23.916 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.915+0000 7fa770ff9640 1 -- 172.21.15.177:0/2868016673 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2348a56d-1908-4fae-9b45-53aaf09fc42a", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fa7741e92f0 con 0x7fa7740b8f30 2024-04-25T15:19:23.916 INFO:teuthology.orchestra.run.smithi177.stdout:6 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 msgr2=0x7fa77409f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa77409f250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00c340 tx=0x7fa75c00c800 comp rx=0 tx=0).stop 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 shutdown_connections 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7741e89a0 0x7fa7741e8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7740b8f30 0x7fa77409f250 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 --2- 172.21.15.177:0/2868016673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7740b4ca0 0x7fa7741d6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 >> 172.21.15.177:0/2868016673 conn(0x7fa7740b5430 msgr2=0x7fa774071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 shutdown_connections 2024-04-25T15:19:23.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:23.916+0000 7fa779d7a640 1 -- 172.21.15.177:0/2868016673 wait complete. 2024-04-25T15:19:23.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 339377f9-bb6b-4cd9-b892-2b0f7bb03df5 7 2024-04-25T15:19:24.040 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.039+0000 7fac5e1ee640 1 Processor -- start 2024-04-25T15:19:24.040 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.039+0000 7fac5e1ee640 1 -- start start 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac5e1ee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac5809bde0 0x7fac580d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac5e1ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5815d2e0 con 0x7fac5815ced0 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac5e1ee640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac58161850 con 0x7fac580d2da0 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac5e1ee640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fac5806f360 con 0x7fac5809bde0 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac57fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac5809bde0 0x7fac580d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac56ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fac580d2da0 0x7fac580d0990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:50114/0 (socket says 172.21.15.177:50114) 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac57fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac5809bde0 0x7fac580d10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53656/0 (socket says 172.21.15.177:53656) 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac56ffd640 1 -- 172.21.15.177:0/2219775859 learned_addr learned my addr 172.21.15.177:0/2219775859 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac57fff640 1 -- 172.21.15.177:0/2219775859 >> v1:172.21.15.177:6789/0 conn(0x7fac580d2da0 legacy=0x7fac580d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac57fff640 1 -- 172.21.15.177:0/2219775859 >> v1:172.21.15.146:6789/0 conn(0x7fac5815ced0 legacy=0x7fac580d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.040+0000 7fac57fff640 1 -- 172.21.15.177:0/2219775859 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5806f640 con 0x7fac5809bde0 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.041+0000 7fac57fff640 1 --2- 172.21.15.177:0/2219775859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809bde0 0x7fac580d10a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac50009d90 tx=0x7fac5002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64c29345319c908b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac567fc640 1 -- 172.21.15.177:0/2219775859 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50007740 con 0x7fac5809bde0 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac567fc640 1 -- 172.21.15.177:0/2219775859 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac500078e0 con 0x7fac5809bde0 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac567fc640 1 -- 172.21.15.177:0/2219775859 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50007cb0 con 0x7fac5809bde0 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac5e1ee640 1 -- 172.21.15.177:0/2219775859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809bde0 msgr2=0x7fac580d10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac5e1ee640 1 --2- 172.21.15.177:0/2219775859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809bde0 0x7fac580d10a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fac50009d90 tx=0x7fac5002a6f0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac5e1ee640 1 -- 172.21.15.177:0/2219775859 shutdown_connections 2024-04-25T15:19:24.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac5e1ee640 1 --2- 172.21.15.177:0/2219775859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809bde0 0x7fac580d10a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac5e1ee640 1 -- 172.21.15.177:0/2219775859 >> 172.21.15.177:0/2219775859 conn(0x7fac580b7430 msgr2=0x7fac580b7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:24.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.042+0000 7fac5e1ee640 1 -- 172.21.15.177:0/2219775859 shutdown_connections 2024-04-25T15:19:24.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.043+0000 7fac5e1ee640 1 -- 172.21.15.177:0/2219775859 wait complete. 2024-04-25T15:19:24.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.044+0000 7fac5e1ee640 1 Processor -- start 2024-04-25T15:19:24.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.044+0000 7fac5e1ee640 1 -- start start 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac5e1ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac5809bde0 0x7fac5806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac5e1ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 0x7fac5814fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac5e1ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 0x7fac5815e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac5e1ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac5809b8b0 con 0x7fac5809bde0 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac5e1ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac5809c1f0 con 0x7fac5809f7a0 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac5e1ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fac581613c0 con 0x7fac5815ced0 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 0x7fac5815e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac56ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 0x7fac5814fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 0x7fac5815e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53660/0 (socket says 172.21.15.177:53660) 2024-04-25T15:19:24.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 -- 172.21.15.177:0/3354320373 learned_addr learned my addr 172.21.15.177:0/3354320373 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac56ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 0x7fac5814fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:54448/0 (socket says 172.21.15.177:54448) 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac577fe640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac5809bde0 0x7fac5806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 -- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 msgr2=0x7fac5814fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 0x7fac5814fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 -- 172.21.15.177:0/3354320373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac5809bde0 msgr2=0x7fac5806e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac5809bde0 0x7fac5806e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 -- 172.21.15.177:0/3354320373 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac581501b0 con 0x7fac5809f7a0 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac56ffd640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 0x7fac5814fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.046+0000 7fac57fff640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 0x7fac5815e200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac50009e80 tx=0x7fac50004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50031cc0 con 0x7fac5809f7a0 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac5003b410 con 0x7fac5809f7a0 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5806d320 con 0x7fac5809f7a0 2024-04-25T15:19:24.052 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50031e60 con 0x7fac5809f7a0 2024-04-25T15:19:24.053 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5806d8e0 con 0x7fac5809f7a0 2024-04-25T15:19:24.053 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fac5003b7e0 con 0x7fac5809f7a0 2024-04-25T15:19:24.053 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.047+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fac5004c3f0 con 0x7fac5809f7a0 2024-04-25T15:19:24.053 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.048+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac580b34b0 con 0x7fac5809f7a0 2024-04-25T15:19:24.053 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.050+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fac580b34b0 con 0x7fac5809f7a0 2024-04-25T15:19:24.093 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:24.092+0000 7f8fe8123040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:19:24.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.122+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "339377f9-bb6b-4cd9-b892-2b0f7bb03df5", "id": 7} v 0) -- 0x7fac5806a0e0 con 0x7fac5809f7a0 2024-04-25T15:19:24.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.126+0000 7fac3ffff640 1 -- 172.21.15.177:0/3354320373 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "339377f9-bb6b-4cd9-b892-2b0f7bb03df5", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fac581501b0 con 0x7fac5809f7a0 2024-04-25T15:19:24.128 INFO:teuthology.orchestra.run.smithi177.stdout:7 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 msgr2=0x7fac5815e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 0x7fac5815e200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac50009e80 tx=0x7fac50004a40 comp rx=0 tx=0).stop 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 shutdown_connections 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac5809f7a0 0x7fac5815e200 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5815ced0 0x7fac5814fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 --2- 172.21.15.177:0/3354320373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac5809bde0 0x7fac5806e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 >> 172.21.15.177:0/3354320373 conn(0x7fac580b7430 msgr2=0x7fac580705d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 shutdown_connections 2024-04-25T15:19:24.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:19:24.127+0000 7fac5e1ee640 1 -- 172.21.15.177:0/3354320373 wait complete. 2024-04-25T15:19:24.139 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T15:19:24.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T15:19:24.142 INFO:tasks.ceph.osd.0:Started 2024-04-25T15:19:24.142 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T15:19:24.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T15:19:24.144 INFO:tasks.ceph.osd.1:Started 2024-04-25T15:19:24.144 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T15:19:24.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T15:19:24.146 INFO:tasks.ceph.osd.2:Started 2024-04-25T15:19:24.146 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T15:19:24.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T15:19:24.148 INFO:tasks.ceph.osd.3:Started 2024-04-25T15:19:24.149 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T15:19:24.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T15:19:24.150 INFO:tasks.ceph.osd.4:Started 2024-04-25T15:19:24.151 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T15:19:24.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T15:19:24.153 INFO:tasks.ceph.osd.5:Started 2024-04-25T15:19:24.153 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T15:19:24.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T15:19:24.155 INFO:tasks.ceph.osd.6:Started 2024-04-25T15:19:24.155 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T15:19:24.155 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T15:19:24.157 INFO:tasks.ceph.osd.7:Started 2024-04-25T15:19:24.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:24.267 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:24.265+0000 7fdf75633040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:19:24.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.295+0000 7f2813cd8640 1 Processor -- start 2024-04-25T15:19:24.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.295+0000 7f2813cd8640 1 -- start start 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.296+0000 7f2813cd8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 0x7f280c0d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.296+0000 7f2813cd8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f280c0d0870 0x7f280c0c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.296+0000 7f2813cd8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c0cfa90 con 0x7f280c06f120 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.296+0000 7f2813cd8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f280c0d0050 con 0x7f280c0d0870 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.296+0000 7f2813cd8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f280c09bfb0 con 0x7f280c09b2b0 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 0x7f280c0d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 0x7f280c0d0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:58410/0 (socket says 172.21.15.146:58410) 2024-04-25T15:19:24.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 -- 172.21.15.146:0/2948024684 learned_addr learned my addr 172.21.15.146:0/2948024684 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2812275640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f280c0d0870 0x7f280c0c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811a74640 1 --1- 172.21.15.146:0/2948024684 >> v1:172.21.15.177:6789/0 conn(0x7f280c06f120 0x7f280c0b3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:43486/0 (socket says 172.21.15.146:43486) 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 -- 172.21.15.146:0/2948024684 >> v2:172.21.15.177:3300/0 conn(0x7f280c0d0870 msgr2=0x7f280c0c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 --2- 172.21.15.146:0/2948024684 >> v2:172.21.15.177:3300/0 conn(0x7f280c0d0870 0x7f280c0c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 -- 172.21.15.146:0/2948024684 >> v1:172.21.15.177:6789/0 conn(0x7f280c06f120 legacy=0x7f280c0b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2811273640 1 -- 172.21.15.146:0/2948024684 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c0cfd70 con 0x7f280c09b2b0 2024-04-25T15:19:24.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.297+0000 7f2810a72640 1 -- 172.21.15.146:0/2948024684 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316224588 0 0) 0x7f280c0cfa90 con 0x7f280c06f120 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.298+0000 7f2811273640 1 --2- 172.21.15.146:0/2948024684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 0x7f280c0d0330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009870 tx=0x7f27fc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ddb0c8578bab1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.298+0000 7f2810a72640 1 -- 172.21.15.146:0/2948024684 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc037070 con 0x7f280c09b2b0 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.298+0000 7f2810a72640 1 -- 172.21.15.146:0/2948024684 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27fc032440 con 0x7f280c09b2b0 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.298+0000 7f2810a72640 1 -- 172.21.15.146:0/2948024684 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc032810 con 0x7f280c09b2b0 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 -- 172.21.15.146:0/2948024684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 msgr2=0x7f280c0d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2948024684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 0x7f280c0d0330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009870 tx=0x7f27fc02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 -- 172.21.15.146:0/2948024684 shutdown_connections 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2948024684 >> v2:172.21.15.177:3300/0 conn(0x7f280c0d0870 0x7f280c0c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2948024684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c09b2b0 0x7f280c0d0330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 -- 172.21.15.146:0/2948024684 >> 172.21.15.146:0/2948024684 conn(0x7f280c0b7530 msgr2=0x7f280c0b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 -- 172.21.15.146:0/2948024684 shutdown_connections 2024-04-25T15:19:24.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.299+0000 7f2813cd8640 1 -- 172.21.15.146:0/2948024684 wait complete. 2024-04-25T15:19:24.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.301+0000 7f2813cd8640 1 Processor -- start 2024-04-25T15:19:24.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.301+0000 7f2813cd8640 1 -- start start 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2813cd8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 0x7f280c161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2813cd8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f280c09b2b0 0x7f280c1547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2813cd8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c0d0870 0x7f280c0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2813cd8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f280c0b4120 con 0x7f280c06f120 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811a74640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 0x7f280c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2813cd8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f280c0d0c80 con 0x7f280c09b2b0 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811a74640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 0x7f280c161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45356/0 (socket says 172.21.15.146:45356) 2024-04-25T15:19:24.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811a74640 1 -- 172.21.15.146:0/2465071660 learned_addr learned my addr 172.21.15.146:0/2465071660 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f280c09b6c0 con 0x7f280c0d0870 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f280c09b2b0 0x7f280c1547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2812275640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c0d0870 0x7f280c0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 -- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c0d0870 msgr2=0x7f280c0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c0d0870 0x7f280c0c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 -- 172.21.15.146:0/2465071660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 msgr2=0x7f280c161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 0x7f280c161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 -- 172.21.15.146:0/2465071660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c160640 con 0x7f280c09b2b0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811a74640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 0x7f280c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.303+0000 7f2811273640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f280c09b2b0 0x7f280c1547e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f280c09c130 tx=0x7f27fc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc037040 con 0x7f280c09b2b0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27fc004230 con 0x7f280c09b2b0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280c160920 con 0x7f280c09b2b0 2024-04-25T15:19:24.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc003610 con 0x7f280c09b2b0 2024-04-25T15:19:24.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280c154d20 con 0x7f280c09b2b0 2024-04-25T15:19:24.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f27fc02ad10 con 0x7f280c09b2b0 2024-04-25T15:19:24.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.304+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f27fc046ad0 con 0x7f280c09b2b0 2024-04-25T15:19:24.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.305+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f280c06a0e0 con 0x7f280c09b2b0 2024-04-25T15:19:24.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.309+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f27fc003d70 con 0x7f280c09b2b0 2024-04-25T15:19:24.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.370+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f280c0c9770 con 0x7f280c09b2b0 2024-04-25T15:19:24.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.371+0000 7f28027fc640 1 -- 172.21.15.146:0/2465071660 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f27fc046d70 con 0x7f280c09b2b0 2024-04-25T15:19:24.373 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:24.373 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":10,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:24.124526+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f280c09b2b0 msgr2=0x7f280c1547e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f280c09b2b0 0x7f280c1547e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f280c09c130 tx=0x7f27fc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 shutdown_connections 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f280c0d0870 0x7f280c0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f280c09b2b0 0x7f280c1547e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 --2- 172.21.15.146:0/2465071660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280c06f120 0x7f280c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 >> 172.21.15.146:0/2465071660 conn(0x7f280c0b7530 msgr2=0x7f280c0ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 shutdown_connections 2024-04-25T15:19:24.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:24.372+0000 7f2813cd8640 1 -- 172.21.15.146:0/2465071660 wait complete. 2024-04-25T15:19:24.385 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T15:19:24.385 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T15:19:24.401 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:24.399+0000 7f8fe8123040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:19:24.453 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:24.451+0000 7f8fe8123040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:19:24.528 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:24.526+0000 7fdf75633040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:19:24.571 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:24.569+0000 7f8fe8123040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:19:24.581 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:24.580+0000 7fdf75633040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:19:24.697 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:24.696+0000 7fdf75633040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:19:24.770 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-04-25T15:19:24.768+0000 7fa27bc41640 -1 Falling back to public interface 2024-04-25T15:19:24.775 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-04-25T15:19:24.773+0000 7f641adcb640 -1 Falling back to public interface 2024-04-25T15:19:24.780 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-04-25T15:19:24.778+0000 7fda4ea0e640 -1 Falling back to public interface 2024-04-25T15:19:24.783 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-04-25T15:19:24.782+0000 7f91b5e61640 -1 Falling back to public interface 2024-04-25T15:19:24.786 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-04-25T15:19:24.785+0000 7fa629eff640 -1 Falling back to public interface 2024-04-25T15:19:24.787 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-04-25T15:19:24.785+0000 7f42c00e5640 -1 Falling back to public interface 2024-04-25T15:19:24.788 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-04-25T15:19:24.786+0000 7f0bf0fc0640 -1 Falling back to public interface 2024-04-25T15:19:24.793 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-04-25T15:19:24.791+0000 7f0b227a7640 -1 Falling back to public interface 2024-04-25T15:19:24.852 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:24.850+0000 7f8fe8123040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:19:24.900 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:24.898+0000 7f8fe8123040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:19:24.982 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:24.980+0000 7fdf75633040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:19:25.029 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.028+0000 7fdf75633040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:19:25.058 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.056+0000 7f8fe8123040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:19:25.187 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.186+0000 7fdf75633040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:19:25.190 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:25.195 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.193+0000 7f8fe8123040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:19:25.248 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.247+0000 7f8fe8123040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:19:25.292 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.290+0000 7f42072e0640 1 Processor -- start 2024-04-25T15:19:25.292 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.290+0000 7f42072e0640 1 -- start start 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f42072e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42000cfb60 0x7f42000b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f42072e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42000b3f20 con 0x7f42000b3b10 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f42072e0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42000b1f60 con 0x7f42000b1870 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f42072e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f42000d08a0 con 0x7f42000cfb60 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f420487b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42000cfb60 0x7f42000b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f420587d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f42000b3b10 0x7f42000cdd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34968/0 (socket says 172.21.15.146:34968) 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f420587d640 1 -- 172.21.15.146:0/2063655597 learned_addr learned my addr 172.21.15.146:0/2063655597 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:25.294 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f420487b640 1 -- 172.21.15.146:0/2063655597 >> v1:172.21.15.177:6789/0 conn(0x7f42000b1870 legacy=0x7f42000b7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:25.295 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f420487b640 1 -- 172.21.15.146:0/2063655597 >> v1:172.21.15.146:6789/0 conn(0x7f42000b3b10 legacy=0x7f42000cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:25.295 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f41f7fff640 1 -- 172.21.15.146:0/2063655597 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225670737 0 0) 0x7f42000b1f60 con 0x7f42000b1870 2024-04-25T15:19:25.295 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.292+0000 7f420487b640 1 -- 172.21.15.146:0/2063655597 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42000b2240 con 0x7f42000cfb60 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.293+0000 7f420487b640 1 --2- 172.21.15.146:0/2063655597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000cfb60 0x7f42000b35d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097a0 tx=0x7f41fc02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db692c903e3a93b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.293+0000 7f41f7fff640 1 -- 172.21.15.146:0/2063655597 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc037070 con 0x7f42000cfb60 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.293+0000 7f41f7fff640 1 -- 172.21.15.146:0/2063655597 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41fc02adc0 con 0x7f42000cfb60 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.293+0000 7f41f7fff640 1 -- 172.21.15.146:0/2063655597 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc032660 con 0x7f42000cfb60 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 -- 172.21.15.146:0/2063655597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000cfb60 msgr2=0x7f42000b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 --2- 172.21.15.146:0/2063655597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000cfb60 0x7f42000b35d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097a0 tx=0x7f41fc02a130 comp rx=0 tx=0).stop 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 -- 172.21.15.146:0/2063655597 shutdown_connections 2024-04-25T15:19:25.296 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 --2- 172.21.15.146:0/2063655597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000cfb60 0x7f42000b35d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:25.297 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 -- 172.21.15.146:0/2063655597 >> 172.21.15.146:0/2063655597 conn(0x7f420008c840 msgr2=0x7f420009ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:25.297 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 -- 172.21.15.146:0/2063655597 shutdown_connections 2024-04-25T15:19:25.297 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.294+0000 7f42072e0640 1 -- 172.21.15.146:0/2063655597 wait complete. 2024-04-25T15:19:25.298 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.296+0000 7f42072e0640 1 Processor -- start 2024-04-25T15:19:25.298 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.296+0000 7f42072e0640 1 -- start start 2024-04-25T15:19:25.301 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f42072e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 0x7f42001eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:25.301 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f42072e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 0x7f420009fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:25.301 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f42072e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 0x7f42000713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:25.301 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f42072e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42000b2800 con 0x7f42000b3b10 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f42072e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42000cff70 con 0x7f42000b1870 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f42072e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f42000b2ae0 con 0x7f42000cfb60 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420487b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 0x7f420009fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420587d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 0x7f42000713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420507c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 0x7f42001eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420507c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 0x7f42001eb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47268/0 (socket says 172.21.15.146:47268) 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420487b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 0x7f420009fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45374/0 (socket says 172.21.15.146:45374) 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420587d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 0x7f42000713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:58440/0 (socket says 172.21.15.146:58440) 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.297+0000 7f420507c640 1 -- 172.21.15.146:0/2059673271 learned_addr learned my addr 172.21.15.146:0/2059673271 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420587d640 1 -- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 msgr2=0x7f42001eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420587d640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 0x7f42001eb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420587d640 1 -- 172.21.15.146:0/2059673271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 msgr2=0x7f420009fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420587d640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 0x7f420009fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:25.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420587d640 1 -- 172.21.15.146:0/2059673271 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42000a0000 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420507c640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 0x7f42001eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420487b640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 0x7f420009fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f420587d640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 0x7f42000713d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41f800cac0 tx=0x7f41f800cf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8018070 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f800a760 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42001e9ae0 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800ab30 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42001ea0a0 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.298+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f41f801b070 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.299+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f41f800acd0 con 0x7f42000cfb60 2024-04-25T15:19:25.303 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.300+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4200050740 con 0x7f42000cfb60 2024-04-25T15:19:25.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.302+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f42000b3f20 con 0x7f42000cfb60 2024-04-25T15:19:25.324 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.323+0000 7f8fe8123040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:19:25.325 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.324+0000 7fdf75633040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:19:25.365 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.363+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42001e9800 con 0x7f42000cfb60 2024-04-25T15:19:25.366 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.364+0000 7f41f5ffb640 1 -- 172.21.15.146:0/2059673271 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f41f80070d0 con 0x7f42000cfb60 2024-04-25T15:19:25.366 INFO:teuthology.misc.health.smithi146.stdout: 2024-04-25T15:19:25.366 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":10,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:24.124526+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:25.367 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 msgr2=0x7f42000713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:25.367 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 0x7f42000713d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41f800cac0 tx=0x7f41f800cf80 comp rx=0 tx=0).stop 2024-04-25T15:19:25.367 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 shutdown_connections 2024-04-25T15:19:25.367 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42000cfb60 0x7f42000713d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:25.367 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42000b3b10 0x7f420009fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:25.368 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 --2- 172.21.15.146:0/2059673271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42000b1870 0x7f42001eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:25.368 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 >> 172.21.15.146:0/2059673271 conn(0x7f420008c840 msgr2=0x7f42000b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:25.368 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 shutdown_connections 2024-04-25T15:19:25.368 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:25.365+0000 7f42072e0640 1 -- 172.21.15.146:0/2059673271 wait complete. 2024-04-25T15:19:25.372 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.370+0000 7f8fe8123040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:19:25.377 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T15:19:25.377 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.375+0000 7fdf75633040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:19:25.446 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.444+0000 7fdf75633040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:19:25.491 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.489+0000 7fdf75633040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:19:25.857 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.855+0000 7f8fe8123040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:19:25.973 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:25.971+0000 7fdf75633040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:19:25.974 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:25.973+0000 7f8fe8123040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:19:26.093 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-04-25T15:19:26.092+0000 7fdf75633040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:19:26.989 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:26.987+0000 7f8f882d3640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T15:19:26.989 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:19:26.988+0000 7f8f882d3640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T15:19:28.248 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-04-25T15:19:28.246+0000 7fa27bc41640 -1 osd.6 0 log_to_monitors true 2024-04-25T15:19:28.254 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-04-25T15:19:28.253+0000 7f641adcb640 -1 osd.5 0 log_to_monitors true 2024-04-25T15:19:28.289 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-04-25T15:19:28.288+0000 7fa629eff640 -1 osd.7 0 log_to_monitors true 2024-04-25T15:19:28.293 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-04-25T15:19:28.291+0000 7f91b5e61640 -1 osd.3 0 log_to_monitors true 2024-04-25T15:19:28.299 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-04-25T15:19:28.298+0000 7f42c00e5640 -1 osd.4 0 log_to_monitors true 2024-04-25T15:19:28.302 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-04-25T15:19:28.301+0000 7f0b227a7640 -1 osd.1 0 log_to_monitors true 2024-04-25T15:19:28.312 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-04-25T15:19:28.310+0000 7f0bf0fc0640 -1 osd.0 0 log_to_monitors true 2024-04-25T15:19:28.312 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-04-25T15:19:28.311+0000 7fda4ea0e640 -1 osd.2 0 log_to_monitors true 2024-04-25T15:19:31.753 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-04-25T15:19:31.751+0000 7fa625e95640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T15:19:31.769 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-04-25T15:19:31.767+0000 7fa61dc90640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:31.976 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-04-25T15:19:31.974+0000 7fa277bd7640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T15:19:31.993 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-04-25T15:19:31.992+0000 7fa26f9d2640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:31.997 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-04-25T15:19:31.995+0000 7f42bc07b640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T15:19:32.008 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-04-25T15:19:32.007+0000 7f42b3e76640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:32.144 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-04-25T15:19:32.142+0000 7f6416d61640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T15:19:32.164 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-04-25T15:19:32.163+0000 7f640eb5c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:32.176 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-04-25T15:19:32.175+0000 7f91b1df7640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T15:19:32.182 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:32.194 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-04-25T15:19:32.192+0000 7f91a9bf2640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:32.223 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-04-25T15:19:32.220+0000 7f0b1e73d640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T15:19:32.226 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-04-25T15:19:32.225+0000 7f0becf56640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T15:19:32.228 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-04-25T15:19:32.227+0000 7fda4a3ce640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T15:19:32.234 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-04-25T15:19:32.233+0000 7f0b16538640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:32.240 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-04-25T15:19:32.239+0000 7f0be4d51640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:32.247 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-04-25T15:19:32.246+0000 7fda421c9640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:19:32.301 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.300+0000 7f167fad8640 1 Processor -- start 2024-04-25T15:19:32.302 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.300+0000 7f167fad8640 1 -- start start 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167fad8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1678071f50 0x7f16780b2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167fad8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16780b74d0 0x7f16780b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167fad8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16781618a0 con 0x7f16780d28f0 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167fad8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f16780b9760 con 0x7f1678071f50 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167fad8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f16780cea40 con 0x7f16780b74d0 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167d874640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1678071f50 0x7f16780b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167d874640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1678071f50 0x7f16780b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36830/0 (socket says 172.21.15.146:36830) 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167d874640 1 -- 172.21.15.146:0/377530178 learned_addr learned my addr 172.21.15.146:0/377530178 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167e075640 1 --2- 172.21.15.146:0/377530178 >> v2:172.21.15.177:3300/0 conn(0x7f16780b74d0 0x7f16780b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:32.304 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.301+0000 7f167d874640 1 -- 172.21.15.146:0/377530178 >> v2:172.21.15.177:3300/0 conn(0x7f16780b74d0 msgr2=0x7f16780b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167d874640 1 --2- 172.21.15.146:0/377530178 >> v2:172.21.15.177:3300/0 conn(0x7f16780b74d0 0x7f16780b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167d874640 1 -- 172.21.15.146:0/377530178 >> v1:172.21.15.177:6789/0 conn(0x7f16780d28f0 legacy=0x7f16780d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167d874640 1 -- 172.21.15.146:0/377530178 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16780ced20 con 0x7f1678071f50 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167e075640 1 --2- 172.21.15.146:0/377530178 >> v2:172.21.15.177:3300/0 conn(0x7f16780b74d0 0x7f16780b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167d874640 1 --2- 172.21.15.146:0/377530178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1678071f50 0x7f16780b2970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16780ca860 tx=0x7f167402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9182970ce30db18f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167c872640 1 -- 172.21.15.146:0/377530178 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674036070 con 0x7f1678071f50 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167c872640 1 -- 172.21.15.146:0/377530178 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1674031600 con 0x7f1678071f50 2024-04-25T15:19:32.305 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.302+0000 7f167c872640 1 -- 172.21.15.146:0/377530178 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16740319d0 con 0x7f1678071f50 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 -- 172.21.15.146:0/377530178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1678071f50 msgr2=0x7f16780b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 --2- 172.21.15.146:0/377530178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1678071f50 0x7f16780b2970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f16780ca860 tx=0x7f167402a730 comp rx=0 tx=0).stop 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 -- 172.21.15.146:0/377530178 shutdown_connections 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 --2- 172.21.15.146:0/377530178 >> v2:172.21.15.177:3300/0 conn(0x7f16780b74d0 0x7f16780b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 --2- 172.21.15.146:0/377530178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1678071f50 0x7f16780b2970 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 -- 172.21.15.146:0/377530178 >> 172.21.15.146:0/377530178 conn(0x7f16780d3800 msgr2=0x7f16780d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 -- 172.21.15.146:0/377530178 shutdown_connections 2024-04-25T15:19:32.306 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.303+0000 7f167fad8640 1 -- 172.21.15.146:0/377530178 wait complete. 2024-04-25T15:19:32.307 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.305+0000 7f167fad8640 1 Processor -- start 2024-04-25T15:19:32.307 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.305+0000 7f167fad8640 1 -- start start 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167fad8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678071f50 0x7f16781fa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167fad8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780b74d0 0x7f1678150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167fad8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 0x7f1678157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167fad8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f16780ce5b0 con 0x7f16780d28f0 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167fad8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f16780cf2e0 con 0x7f16780b74d0 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167fad8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f16780ce730 con 0x7f1678071f50 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167e075640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 0x7f1678157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167e075640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 0x7f1678157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36840/0 (socket says 172.21.15.146:36840) 2024-04-25T15:19:32.309 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167e075640 1 -- 172.21.15.146:0/4264022800 learned_addr learned my addr 172.21.15.146:0/4264022800 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167d073640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780b74d0 0x7f1678150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167d874640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678071f50 0x7f16781fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.306+0000 7f167e075640 1 -- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678071f50 msgr2=0x7f16781fa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167e075640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678071f50 0x7f16781fa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167e075640 1 -- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780b74d0 msgr2=0x7f1678150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167e075640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780b74d0 0x7f1678150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167e075640 1 -- 172.21.15.146:0/4264022800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167814c990 con 0x7f16780d28f0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167e075640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 0x7f1678157b60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f167000d5b0 tx=0x7f167000da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670018070 con 0x7f16780d28f0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1670004030 con 0x7f16780d28f0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167814cc70 con 0x7f16780d28f0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670004400 con 0x7f16780d28f0 2024-04-25T15:19:32.310 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.307+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167814d230 con 0x7f16780d28f0 2024-04-25T15:19:32.311 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.308+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1678075230 con 0x7f16780d28f0 2024-04-25T15:19:32.311 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.308+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 4) ==== 96726+0+0 (secure 0 0 0) 0x7f1670014620 con 0x7f16780d28f0 2024-04-25T15:19:32.311 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.309+0000 7f166e7fc640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f165c074420 0x7f165c076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:32.312 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.309+0000 7f167d874640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f165c074420 0x7f165c076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:32.312 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.309+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f1670094550 con 0x7f16780d28f0 2024-04-25T15:19:32.312 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.309+0000 7f167d874640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f165c074420 0x7f165c076930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1674033000 tx=0x7f1674009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:32.315 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.314+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f167005f500 con 0x7f16780d28f0 2024-04-25T15:19:32.490 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.489+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16780b40c0 con 0x7f16780d28f0 2024-04-25T15:19:32.491 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.489+0000 7f166e7fc640 1 -- 172.21.15.146:0/4264022800 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f1670063800 con 0x7f16780d28f0 2024-04-25T15:19:32.491 INFO:teuthology.misc.health.smithi146.stdout: 2024-04-25T15:19:32.491 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":12,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:29.992045+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:32.493 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f165c074420 msgr2=0x7f165c076930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f165c074420 0x7f165c076930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1674033000 tx=0x7f1674009000 comp rx=0 tx=0).stop 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 msgr2=0x7f1678157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 0x7f1678157b60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f167000d5b0 tx=0x7f167000da70 comp rx=0 tx=0).stop 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 shutdown_connections 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f165c074420 0x7f165c076930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780d28f0 0x7f1678157b60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780b74d0 0x7f1678150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 --2- 172.21.15.146:0/4264022800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678071f50 0x7f16781fa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.491+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 >> 172.21.15.146:0/4264022800 conn(0x7f16780d3800 msgr2=0x7f16780b8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.492+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 shutdown_connections 2024-04-25T15:19:32.494 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:32.492+0000 7f167fad8640 1 -- 172.21.15.146:0/4264022800 wait complete. 2024-04-25T15:19:32.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T15:19:39.310 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:39.424 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.422+0000 7fe142935640 1 Processor -- start 2024-04-25T15:19:39.424 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.422+0000 7fe142935640 1 -- start start 2024-04-25T15:19:39.426 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.423+0000 7fe142935640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe13c0d05a0 0x7fe13c0b7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.426 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.423+0000 7fe142935640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0b19e0 0x7fe13c0b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.426 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.423+0000 7fe142935640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe13c0d26d0 con 0x7fe13c06f4d0 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.423+0000 7fe142935640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe13c0d29b0 con 0x7fe13c0d05a0 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.423+0000 7fe142935640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe13c0d2c90 con 0x7fe13c0b19e0 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe13c0d05a0 0x7fe13c0b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe13c0d05a0 0x7fe13c0b7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44136/0 (socket says 172.21.15.146:44136) 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13b7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe13c06f4d0 0x7fe13c06def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:54334/0 (socket says 172.21.15.146:54334) 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 -- 172.21.15.146:0/2268509320 learned_addr learned my addr 172.21.15.146:0/2268509320 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe140ed2640 1 --2- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0b19e0 0x7fe13c0b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 -- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0b19e0 msgr2=0x7fe13c0b79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 --2- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0b19e0 0x7fe13c0b79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 -- 172.21.15.146:0/2268509320 >> v1:172.21.15.177:6789/0 conn(0x7fe13c06f4d0 legacy=0x7fe13c06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.424+0000 7fe13bfff640 1 -- 172.21.15.146:0/2268509320 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c0d2f70 con 0x7fe13c0d05a0 2024-04-25T15:19:39.427 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.425+0000 7fe13bfff640 1 --2- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0d05a0 0x7fe13c0b7480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009870 tx=0x7fe12c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a127d32ff6bb353 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:39.429 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.427+0000 7fe13affd640 1 -- 172.21.15.146:0/2268509320 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c037070 con 0x7fe13c0d05a0 2024-04-25T15:19:39.429 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.427+0000 7fe13affd640 1 -- 172.21.15.146:0/2268509320 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12c032440 con 0x7fe13c0d05a0 2024-04-25T15:19:39.429 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.427+0000 7fe13affd640 1 -- 172.21.15.146:0/2268509320 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c032810 con 0x7fe13c0d05a0 2024-04-25T15:19:39.430 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 -- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0d05a0 msgr2=0x7fe13c0b7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.430 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 --2- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0d05a0 0x7fe13c0b7480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009870 tx=0x7fe12c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.430 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 -- 172.21.15.146:0/2268509320 shutdown_connections 2024-04-25T15:19:39.430 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 --2- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0b19e0 0x7fe13c0b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.431 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 --2- 172.21.15.146:0/2268509320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0d05a0 0x7fe13c0b7480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.431 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 -- 172.21.15.146:0/2268509320 >> 172.21.15.146:0/2268509320 conn(0x7fe13c08c840 msgr2=0x7fe13c09ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:39.431 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 -- 172.21.15.146:0/2268509320 shutdown_connections 2024-04-25T15:19:39.431 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.428+0000 7fe142935640 1 -- 172.21.15.146:0/2268509320 wait complete. 2024-04-25T15:19:39.432 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.430+0000 7fe142935640 1 Processor -- start 2024-04-25T15:19:39.432 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.430+0000 7fe142935640 1 -- start start 2024-04-25T15:19:39.435 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.431+0000 7fe142935640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 0x7fe13c06ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.435 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.431+0000 7fe142935640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0b19e0 0x7fe13c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.435 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.431+0000 7fe142935640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0d05a0 0x7fe13c158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.435 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.431+0000 7fe142935640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c0d23d0 con 0x7fe13c06f4d0 2024-04-25T15:19:39.435 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.431+0000 7fe142935640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c0d2250 con 0x7fe13c0b19e0 2024-04-25T15:19:39.435 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.431+0000 7fe142935640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe13c0d1c00 con 0x7fe13c0d05a0 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 0x7fe13c06ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 0x7fe13c06ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36874/0 (socket says 172.21.15.146:36874) 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 -- 172.21.15.146:0/1280315060 learned_addr learned my addr 172.21.15.146:0/1280315060 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe140ed2640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0d05a0 0x7fe13c158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13b7fe640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0b19e0 0x7fe13c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 -- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0d05a0 msgr2=0x7fe13c158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0d05a0 0x7fe13c158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 -- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0b19e0 msgr2=0x7fe13c06d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0b19e0 0x7fe13c06d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 -- 172.21.15.146:0/1280315060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c156230 con 0x7fe13c06f4d0 2024-04-25T15:19:39.436 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe13bfff640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 0x7fe13c06ce10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009870 tx=0x7fe12c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:39.437 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c041020 con 0x7fe13c06f4d0 2024-04-25T15:19:39.437 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12c03d6e0 con 0x7fe13c06f4d0 2024-04-25T15:19:39.437 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c156510 con 0x7fe13c06f4d0 2024-04-25T15:19:39.437 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c037040 con 0x7fe13c06f4d0 2024-04-25T15:19:39.437 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.432+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c15e440 con 0x7fe13c06f4d0 2024-04-25T15:19:39.437 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.433+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13c050740 con 0x7fe13c06f4d0 2024-04-25T15:19:39.438 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.434+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe12c044020 con 0x7fe13c06f4d0 2024-04-25T15:19:39.438 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.435+0000 7fe138ff9640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.438 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.435+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe12c0b56c0 con 0x7fe13c06f4d0 2024-04-25T15:19:39.438 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.435+0000 7fe13b7fe640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.438 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.435+0000 7fe13b7fe640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe13c1533a0 tx=0x7fe130007ce0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:39.441 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.439+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe12c04b030 con 0x7fe13c06f4d0 2024-04-25T15:19:39.615 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.614+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe13c15cd00 con 0x7fe13c06f4d0 2024-04-25T15:19:39.616 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.615+0000 7fe138ff9640 1 -- 172.21.15.146:0/1280315060 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12932 (secure 0 0 0) 0x7fe12c0801b0 con 0x7fe13c06f4d0 2024-04-25T15:19:39.617 INFO:teuthology.misc.health.smithi146.stdout: 2024-04-25T15:19:39.617 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":16,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:36.044209+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 msgr2=0x7fe1280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe13c1533a0 tx=0x7fe130007ce0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 msgr2=0x7fe13c06ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 0x7fe13c06ce10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009870 tx=0x7fe12c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 shutdown_connections 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe13c0d05a0 0x7fe13c158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c0b19e0 0x7fe13c06d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 --2- 172.21.15.146:0/1280315060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe13c06f4d0 0x7fe13c06ce10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.619 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 >> 172.21.15.146:0/1280315060 conn(0x7fe13c08c840 msgr2=0x7fe13c0af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:39.620 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 shutdown_connections 2024-04-25T15:19:39.620 INFO:teuthology.misc.health.smithi146.stderr:2024-04-25T15:19:39.617+0000 7fe142935640 1 -- 172.21.15.146:0/1280315060 wait complete. 2024-04-25T15:19:39.629 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T15:19:39.629 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T15:19:39.629 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:19:39.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:19:39.632 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:19:39.633 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:19:39.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:19:39.635 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:19:39.635 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:19:39.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:19:39.638 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:19:39.638 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:19:39.638 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:19:39.643 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:19:39.643 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T15:19:39.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:39.717 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-04-25T15:19:39.717 INFO:tasks.ceph.mds.a.smithi146.stdout:starting mds.a at 2024-04-25T15:19:39.733 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-04-25T15:19:39.734 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-04-25T15:19:39.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.757+0000 7fa10f51f640 1 Processor -- start 2024-04-25T15:19:39.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.757+0000 7fa10f51f640 1 -- start start 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10f51f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa1080d12a0 0x7fa10815ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10f51f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080b45d0 0x7fa1080d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10f51f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1080d1050 con 0x7fa1080d0c40 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10f51f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa1080d4db0 con 0x7fa1080d12a0 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10f51f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa10806de00 con 0x7fa1080b45d0 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10caba640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080b45d0 0x7fa1080d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10d2bb640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa1080d12a0 0x7fa10815ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10caba640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080b45d0 0x7fa1080d0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:40036/0 (socket says 172.21.15.146:40036) 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10caba640 1 -- 172.21.15.146:0/168370317 learned_addr learned my addr 172.21.15.146:0/168370317 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:39.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10d2bb640 1 -- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080b45d0 msgr2=0x7fa1080d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:39.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10d2bb640 1 --2- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080b45d0 0x7fa1080d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10d2bb640 1 -- 172.21.15.146:0/168370317 >> v1:172.21.15.177:6789/0 conn(0x7fa1080d0c40 legacy=0x7fa108163b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.759+0000 7fa10d2bb640 1 -- 172.21.15.146:0/168370317 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1080d5000 con 0x7fa1080d12a0 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.760+0000 7fa10d2bb640 1 --2- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa10815ff70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097a0 tx=0x7fa10402a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5093b59ce0a32fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.760+0000 7fa0fffff640 1 -- 172.21.15.146:0/168370317 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104037070 con 0x7fa1080d12a0 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.760+0000 7fa0fffff640 1 -- 172.21.15.146:0/168370317 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10402adb0 con 0x7fa1080d12a0 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.760+0000 7fa0fffff640 1 -- 172.21.15.146:0/168370317 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104032660 con 0x7fa1080d12a0 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 -- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 msgr2=0x7fa10815ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 --2- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa10815ff70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097a0 tx=0x7fa10402a060 comp rx=0 tx=0).stop 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 -- 172.21.15.146:0/168370317 shutdown_connections 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 --2- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080b45d0 0x7fa1080d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 --2- 172.21.15.146:0/168370317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa10815ff70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 -- 172.21.15.146:0/168370317 >> 172.21.15.146:0/168370317 conn(0x7fa1080d63b0 msgr2=0x7fa1080d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:39.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 -- 172.21.15.146:0/168370317 shutdown_connections 2024-04-25T15:19:39.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.761+0000 7fa10f51f640 1 -- 172.21.15.146:0/168370317 wait complete. 2024-04-25T15:19:39.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.763+0000 7fa10f51f640 1 Processor -- start 2024-04-25T15:19:39.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.763+0000 7fa10f51f640 1 -- start start 2024-04-25T15:19:39.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10f51f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 0x7fa108157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10f51f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080d0c40 0x7fa1081558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10f51f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa108158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10f51f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa10806c8d0 con 0x7fa1080b45d0 2024-04-25T15:19:39.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10f51f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa10806c5f0 con 0x7fa1080d12a0 2024-04-25T15:19:39.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10f51f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa1080b8f10 con 0x7fa1080d0c40 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10d2bb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 0x7fa108157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10d2bb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 0x7fa108157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36904/0 (socket says 172.21.15.146:36904) 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10d2bb640 1 -- 172.21.15.146:0/1836949045 learned_addr learned my addr 172.21.15.146:0/1836949045 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10caba640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080d0c40 0x7fa1081558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10dabc640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa108158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10caba640 1 -- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 msgr2=0x7fa108158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10caba640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa108158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10caba640 1 -- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 msgr2=0x7fa108157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10caba640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 0x7fa108157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10caba640 1 -- 172.21.15.146:0/1836949045 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10815b5b0 con 0x7fa1080d0c40 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.764+0000 7fa10d2bb640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 0x7fa108157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:39.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.765+0000 7fa10caba640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080d0c40 0x7fa1081558e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800e870 tx=0x7fa0f800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:39.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.765+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8017070 con 0x7fa1080d0c40 2024-04-25T15:19:39.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.765+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f800ee60 con 0x7fa1080d0c40 2024-04-25T15:19:39.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.765+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10815b2d0 con 0x7fa1080d0c40 2024-04-25T15:19:39.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.765+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800cb60 con 0x7fa1080d0c40 2024-04-25T15:19:39.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.765+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1081550e0 con 0x7fa1080d0c40 2024-04-25T15:19:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.766+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa10806a0e0 con 0x7fa1080d0c40 2024-04-25T15:19:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.767+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa0f801a070 con 0x7fa1080d0c40 2024-04-25T15:19:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.767+0000 7fa0fdffb640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0ec0744b0 0x7fa0ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.767+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa0f8094ba0 con 0x7fa1080d0c40 2024-04-25T15:19:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.767+0000 7fa10d2bb640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0ec0744b0 0x7fa0ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.767+0000 7fa10d2bb640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0ec0744b0 0x7fa0ec0769c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa10402a0a0 tx=0x7fa10402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:39.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.773+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0f805f130 con 0x7fa1080d0c40 2024-04-25T15:19:39.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.946+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1080d3e30 con 0x7fa1080d0c40 2024-04-25T15:19:39.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.947+0000 7fa0fdffb640 1 -- 172.21.15.146:0/1836949045 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12932 (secure 0 0 0) 0x7fa0f8063430 con 0x7fa1080d0c40 2024-04-25T15:19:39.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:39.950 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":16,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:36.044209+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0ec0744b0 msgr2=0x7fa0ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0ec0744b0 0x7fa0ec0769c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa10402a0a0 tx=0x7fa10402e000 comp rx=0 tx=0).stop 2024-04-25T15:19:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080d0c40 msgr2=0x7fa1081558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080d0c40 0x7fa1081558e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800e870 tx=0x7fa0f800ed30 comp rx=0 tx=0).stop 2024-04-25T15:19:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 shutdown_connections 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0ec0744b0 0x7fa0ec0769c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1080d12a0 0x7fa108158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1080d0c40 0x7fa1081558e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 --2- 172.21.15.146:0/1836949045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1080b45d0 0x7fa108157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 >> 172.21.15.146:0/1836949045 conn(0x7fa1080d63b0 msgr2=0x7fa10815dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 shutdown_connections 2024-04-25T15:19:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:39.950+0000 7fa10f51f640 1 -- 172.21.15.146:0/1836949045 wait complete. 2024-04-25T15:19:39.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:19:39.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:19:40.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.072+0000 7f772e4aa640 1 Processor -- start 2024-04-25T15:19:40.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.072+0000 7f772e4aa640 1 -- start start 2024-04-25T15:19:40.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772e4aa640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f77280bb450 0x7f77280d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772e4aa640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77280d0de0 0x7f77280b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772e4aa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77280b5230 con 0x7f7728163c00 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772e4aa640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f772806c8c0 con 0x7f77280d0de0 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772e4aa640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f772806cba0 con 0x7f77280bb450 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772ca47640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77280d0de0 0x7f77280b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772ca47640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77280d0de0 0x7f77280b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36916/0 (socket says 172.21.15.146:36916) 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f772ca47640 1 -- 172.21.15.146:0/1483619647 learned_addr learned my addr 172.21.15.146:0/1483619647 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.073+0000 7f7727fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7728163c00 0x7f772806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:58176/0 (socket says 172.21.15.146:58176) 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f77277fe640 1 --2- 172.21.15.146:0/1483619647 >> v2:172.21.15.177:3300/0 conn(0x7f77280bb450 0x7f77280d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f772ca47640 1 -- 172.21.15.146:0/1483619647 >> v2:172.21.15.177:3300/0 conn(0x7f77280bb450 msgr2=0x7f77280d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f772ca47640 1 --2- 172.21.15.146:0/1483619647 >> v2:172.21.15.177:3300/0 conn(0x7f77280bb450 0x7f77280d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f772ca47640 1 -- 172.21.15.146:0/1483619647 >> v1:172.21.15.146:6789/0 conn(0x7f7728163c00 legacy=0x7f772806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f772ca47640 1 -- 172.21.15.146:0/1483619647 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772806ce80 con 0x7f77280d0de0 2024-04-25T15:19:40.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f7726ffd640 1 -- 172.21.15.146:0/1483619647 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422088323 0 0) 0x7f77280b5230 con 0x7f7728163c00 2024-04-25T15:19:40.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.074+0000 7f772ca47640 1 --2- 172.21.15.146:0/1483619647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0de0 0x7f77280b8fb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7720009e50 tx=0x7f772002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66affe81be1f94c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:40.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f7726ffd640 1 -- 172.21.15.146:0/1483619647 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7720036070 con 0x7f77280d0de0 2024-04-25T15:19:40.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f7726ffd640 1 -- 172.21.15.146:0/1483619647 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7720031600 con 0x7f77280d0de0 2024-04-25T15:19:40.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f7726ffd640 1 -- 172.21.15.146:0/1483619647 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77200319d0 con 0x7f77280d0de0 2024-04-25T15:19:40.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 -- 172.21.15.146:0/1483619647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0de0 msgr2=0x7f77280b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1483619647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0de0 0x7f77280b8fb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7720009e50 tx=0x7f772002a6d0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 -- 172.21.15.146:0/1483619647 shutdown_connections 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1483619647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0de0 0x7f77280b8fb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1483619647 >> v2:172.21.15.177:3300/0 conn(0x7f77280bb450 0x7f77280d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 -- 172.21.15.146:0/1483619647 >> 172.21.15.146:0/1483619647 conn(0x7f77280b7970 msgr2=0x7f77280b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 -- 172.21.15.146:0/1483619647 shutdown_connections 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.075+0000 7f772e4aa640 1 -- 172.21.15.146:0/1483619647 wait complete. 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.077+0000 7f772e4aa640 1 Processor -- start 2024-04-25T15:19:40.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.077+0000 7f772e4aa640 1 -- start start 2024-04-25T15:19:40.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 0x7f7728151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77280d0de0 0x7f7728150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7728163c00 0x7f772815ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f77280bb860 con 0x7f77280bb450 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f772806c550 con 0x7f77280d0de0 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f77280bb9e0 con 0x7f7728163c00 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 0x7f7728151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 0x7f7728151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36932/0 (socket says 172.21.15.146:36932) 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 -- 172.21.15.146:0/1086635897 learned_addr learned my addr 172.21.15.146:0/1086635897 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772ca47640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7728163c00 0x7f772815ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f77277fe640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77280d0de0 0x7f7728150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 -- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7728163c00 msgr2=0x7f772815ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7728163c00 0x7f772815ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 -- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77280d0de0 msgr2=0x7f7728150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77280d0de0 0x7f7728150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 -- 172.21.15.146:0/1086635897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77280cb940 con 0x7f77280bb450 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7727fff640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 0x7f7728151a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c460 tx=0x7f771c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c018070 con 0x7f77280bb450 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f771c007500 con 0x7f77280bb450 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7728152390 con 0x7f77280bb450 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c004a20 con 0x7f77280bb450 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.079+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7728152950 con 0x7f77280bb450 2024-04-25T15:19:40.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.080+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77280cbb10 con 0x7f77280bb450 2024-04-25T15:19:40.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.081+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f771c01b070 con 0x7f77280bb450 2024-04-25T15:19:40.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.082+0000 7f7724ff9640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77140744b0 0x7f77140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.082+0000 7f77277fe640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77140744b0 0x7f77140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.082+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f771c0941b0 con 0x7f77280bb450 2024-04-25T15:19:40.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.082+0000 7f77277fe640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77140744b0 0x7f77140769c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7728069c90 tx=0x7f7718008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:40.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.087+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f771c05ec90 con 0x7f77280bb450 2024-04-25T15:19:40.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.261+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f772806a0e0 con 0x7f77280bb450 2024-04-25T15:19:40.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.261+0000 7f7724ff9640 1 -- 172.21.15.146:0/1086635897 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f771c05e590 con 0x7f77280bb450 2024-04-25T15:19:40.263 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:19:40.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77140744b0 msgr2=0x7f77140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77140744b0 0x7f77140769c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7728069c90 tx=0x7f7718008040 comp rx=0 tx=0).stop 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 msgr2=0x7f7728151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 0x7f7728151a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c460 tx=0x7f771c00c920 comp rx=0 tx=0).stop 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 shutdown_connections 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77140744b0 0x7f77140769c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7728163c00 0x7f772815ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77280d0de0 0x7f7728150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 --2- 172.21.15.146:0/1086635897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280bb450 0x7f7728151a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 >> 172.21.15.146:0/1086635897 conn(0x7f77280b7970 msgr2=0x7f772815fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 shutdown_connections 2024-04-25T15:19:40.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.263+0000 7f772e4aa640 1 -- 172.21.15.146:0/1086635897 wait complete. 2024-04-25T15:19:40.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:19:40.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.382+0000 7f760217d640 1 Processor -- start 2024-04-25T15:19:40.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.382+0000 7f760217d640 1 -- start start 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.383+0000 7f760217d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f75fc15cfd0 0x7f75fc0b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.383+0000 7f760217d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75fc0cfff0 con 0x7f75fc0d2c70 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.383+0000 7f760217d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75fc0cf230 con 0x7f75fc09bba0 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.383+0000 7f760217d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f75fc0cefe0 con 0x7f75fc15cfd0 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fbfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f75fc0d2c70 0x7f75fc06def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:58190/0 (socket says 172.21.15.146:58190) 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fbfff640 1 -- 172.21.15.146:0/2197401325 learned_addr learned my addr 172.21.15.146:0/2197401325 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/2197401325 >> v2:172.21.15.177:3300/0 conn(0x7f75fc15cfd0 0x7f75fc0b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617226279 0 0) 0x7f75fc0cfff0 con 0x7f75fc0d2c70 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e4003520 con 0x7f75fc0d2c70 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609871991 0 0) 0x7f75fc0cf230 con 0x7f75fc09bba0 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75fc0cfff0 con 0x7f75fc09bba0 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 203700791 0 0) 0x7f75e4003520 con 0x7f75fc0d2c70 2024-04-25T15:19:40.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 >> v2:172.21.15.177:3300/0 conn(0x7f75fc15cfd0 msgr2=0x7f75fc0b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 --2- 172.21.15.146:0/2197401325 >> v2:172.21.15.177:3300/0 conn(0x7f75fc15cfd0 0x7f75fc0b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 >> v1:172.21.15.177:6789/0 conn(0x7f75fc09bba0 legacy=0x7f75fc0d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc0cf510 con 0x7f75fc0d2c70 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/2197401325 >> v2:172.21.15.177:3300/0 conn(0x7f75fc15cfd0 0x7f75fc0b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f75f4001a10 con 0x7f75fc0d2c70 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 >> v1:172.21.15.146:6789/0 conn(0x7f75fc0d2c70 legacy=0x7f75fc06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75e4003b40 0x7f75e4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75e4004800 0x7f75e4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e40051a0 0x7f75e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc0cefe0 con 0x7f75e4003b40 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75e400c140 con 0x7f75e4004800 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f75e400c340 con 0x7f75e40051a0 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75e4003b40 0x7f75e4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75faffd640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75e4004800 0x7f75e4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fbfff640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e40051a0 0x7f75e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fb7fe640 1 -- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e40051a0 msgr2=0x7f75e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.384+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e40051a0 0x7f75e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fb7fe640 1 -- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75e4004800 msgr2=0x7f75e4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75e4004800 0x7f75e4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fb7fe640 1 -- 172.21.15.146:0/2197401325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e4003520 con 0x7f75e4003b40 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75faffd640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75e4004800 0x7f75e4004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fbfff640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e40051a0 0x7f75e400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fbfff640 1 -- 172.21.15.146:0/2197401325 reap_dead start 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75e4003b40 0x7f75e4003f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75e40057e0 tx=0x7f75e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dcfdfaffdca5c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec036070 con 0x7f75e4003b40 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec031400 con 0x7f75e4003b40 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.385+0000 7f75fa7fc640 1 -- 172.21.15.146:0/2197401325 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec009860 con 0x7f75e4003b40 2024-04-25T15:19:40.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 -- 172.21.15.146:0/2197401325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75e4003b40 msgr2=0x7f75e4003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75e4003b40 0x7f75e4003f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75e40057e0 tx=0x7f75e4005830 comp rx=0 tx=0).stop 2024-04-25T15:19:40.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 -- 172.21.15.146:0/2197401325 shutdown_connections 2024-04-25T15:19:40.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 --2- 172.21.15.146:0/2197401325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75e4003b40 0x7f75e4003f90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 -- 172.21.15.146:0/2197401325 >> 172.21.15.146:0/2197401325 conn(0x7f75fc0b7530 msgr2=0x7f75fc0b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:40.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 -- 172.21.15.146:0/2197401325 shutdown_connections 2024-04-25T15:19:40.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.386+0000 7f760217d640 1 -- 172.21.15.146:0/2197401325 wait complete. 2024-04-25T15:19:40.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.388+0000 7f760217d640 1 Processor -- start 2024-04-25T15:19:40.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.388+0000 7f760217d640 1 -- start start 2024-04-25T15:19:40.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f760217d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e4003b40 0x7f75fc1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f760217d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 0x7f75fc1e7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f760217d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75fc0ccc20 0x7f75fc0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f760217d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc09bfb0 con 0x7f75fc1e7010 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f760217d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc0cfad0 con 0x7f75fc0ccc20 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f760217d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f75fc09c130 con 0x7f75e4003b40 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f75faffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 0x7f75fc1e7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f75faffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 0x7f75fc1e7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36952/0 (socket says 172.21.15.146:36952) 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f75faffd640 1 -- 172.21.15.146:0/1791727404 learned_addr learned my addr 172.21.15.146:0/1791727404 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.389+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e4003b40 0x7f75fc1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75fbfff640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75fc0ccc20 0x7f75fc0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75faffd640 1 -- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e4003b40 msgr2=0x7f75fc1f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75faffd640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e4003b40 0x7f75fc1f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75faffd640 1 -- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75fc0ccc20 msgr2=0x7f75fc0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75faffd640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75fc0ccc20 0x7f75fc0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75faffd640 1 -- 172.21.15.146:0/1791727404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc1e7960 con 0x7f75fc1e7010 2024-04-25T15:19:40.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75faffd640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 0x7f75fc1e7420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e7b0 tx=0x7f75f000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00050d0 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f0005270 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75fc1587c0 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0005420 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.390+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75fc1595b0 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.391+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75fc06a0e0 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.392+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f75f001a020 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.392+0000 7f75e3fff640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75e80744e0 0x7f75e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.392+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75e80744e0 0x7f75e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.392+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f75f0094550 con 0x7f75fc1e7010 2024-04-25T15:19:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.393+0000 7f75fb7fe640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75e80744e0 0x7f75e80769f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75ec033000 tx=0x7f75ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:40.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.397+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75fc1e7960 con 0x7f75fc1e7010 2024-04-25T15:19:40.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.611+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75fc15c110 con 0x7f75fc1e7010 2024-04-25T15:19:40.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.611+0000 7f75e3fff640 1 -- 172.21.15.146:0/1791727404 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f75f005f030 con 0x7f75fc1e7010 2024-04-25T15:19:40.614 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:40.614 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:19:40.614 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 2 2024-04-25T15:19:40.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75e80744e0 msgr2=0x7f75e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75e80744e0 0x7f75e80769f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f75ec033000 tx=0x7f75ec008030 comp rx=0 tx=0).stop 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 msgr2=0x7f75fc1e7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 0x7f75fc1e7420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e7b0 tx=0x7f75f000ebb0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 shutdown_connections 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75e80744e0 0x7f75e80769f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75fc0ccc20 0x7f75fc0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75fc1e7010 0x7f75fc1e7420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 --2- 172.21.15.146:0/1791727404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75e4003b40 0x7f75fc1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:40.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 >> 172.21.15.146:0/1791727404 conn(0x7f75fc0b7530 msgr2=0x7f75fc06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:40.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 shutdown_connections 2024-04-25T15:19:40.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:40.613+0000 7f760217d640 1 -- 172.21.15.146:0/1791727404 wait complete. 2024-04-25T15:19:42.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:19:42.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.748+0000 7f28c59ce640 1 Processor -- start 2024-04-25T15:19:42.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.748+0000 7f28c59ce640 1 -- start start 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28c59ce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c00d4f60 0x7f28c00d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28c59ce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f28c00d0020 0x7f28c00d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28c59ce640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c00b9200 con 0x7f28c0163d80 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28c59ce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f28c006c550 con 0x7f28c00d0020 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28c59ce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f28c006c830 con 0x7f28c00d4f60 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28be7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f28c00d0020 0x7f28c00d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28bf7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f28c0163d80 0x7f28c0164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:57870/0 (socket says 172.21.15.146:57870) 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28be7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f28c00d0020 0x7f28c00d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36036/0 (socket says 172.21.15.146:36036) 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28bf7fe640 1 -- 172.21.15.146:0/1953267529 learned_addr learned my addr 172.21.15.146:0/1953267529 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:42.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28beffd640 1 --2- 172.21.15.146:0/1953267529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c00d4f60 0x7f28c00d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28be7fc640 1 -- 172.21.15.146:0/1953267529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c00d4f60 msgr2=0x7f28c00d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28be7fc640 1 --2- 172.21.15.146:0/1953267529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c00d4f60 0x7f28c00d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28be7fc640 1 -- 172.21.15.146:0/1953267529 >> v1:172.21.15.146:6789/0 conn(0x7f28c0163d80 legacy=0x7f28c0164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28be7fc640 1 -- 172.21.15.146:0/1953267529 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c00d0970 con 0x7f28c00d0020 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.750+0000 7f28bdffb640 1 -- 172.21.15.146:0/1953267529 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096249136 0 0) 0x7f28b8001110 con 0x7f28c0163d80 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.751+0000 7f28be7fc640 1 --2- 172.21.15.146:0/1953267529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c00d0430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28b40097a0 tx=0x7f28b402a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec4e61946ab49bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.751+0000 7f28bdffb640 1 -- 172.21.15.146:0/1953267529 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4037070 con 0x7f28c00d0020 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.751+0000 7f28bdffb640 1 -- 172.21.15.146:0/1953267529 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b402adb0 con 0x7f28c00d0020 2024-04-25T15:19:42.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.751+0000 7f28bdffb640 1 -- 172.21.15.146:0/1953267529 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4032660 con 0x7f28c00d0020 2024-04-25T15:19:42.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 -- 172.21.15.146:0/1953267529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 msgr2=0x7f28c00d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:42.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 --2- 172.21.15.146:0/1953267529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c00d0430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28b40097a0 tx=0x7f28b402a060 comp rx=0 tx=0).stop 2024-04-25T15:19:42.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 -- 172.21.15.146:0/1953267529 shutdown_connections 2024-04-25T15:19:42.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 --2- 172.21.15.146:0/1953267529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c00d0430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 --2- 172.21.15.146:0/1953267529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c00d4f60 0x7f28c00d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 -- 172.21.15.146:0/1953267529 >> 172.21.15.146:0/1953267529 conn(0x7f28c00d2450 msgr2=0x7f28c00d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:42.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 -- 172.21.15.146:0/1953267529 shutdown_connections 2024-04-25T15:19:42.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.752+0000 7f28c59ce640 1 -- 172.21.15.146:0/1953267529 wait complete. 2024-04-25T15:19:42.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.753+0000 7f28c59ce640 1 Processor -- start 2024-04-25T15:19:42.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.754+0000 7f28c59ce640 1 -- start start 2024-04-25T15:19:42.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28c59ce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c015a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:42.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28c59ce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c00d4f60 0x7f28c015d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:42.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28c59ce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c0163d80 0x7f28c015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:42.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28c59ce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f28c00d15f0 con 0x7f28c00d0020 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28c59ce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f28c00d1070 con 0x7f28c00d4f60 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28c59ce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f28c00b5150 con 0x7f28c0163d80 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c015a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c015a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36052/0 (socket says 172.21.15.146:36052) 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 -- 172.21.15.146:0/2253804449 learned_addr learned my addr 172.21.15.146:0/2253804449 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28bf7fe640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c0163d80 0x7f28c015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 -- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c0163d80 msgr2=0x7f28c015ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c0163d80 0x7f28c015ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 -- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c00d4f60 msgr2=0x7f28c015d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c00d4f60 0x7f28c015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28beffd640 1 -- 172.21.15.146:0/2253804449 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c01ea590 con 0x7f28c00d0020 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.755+0000 7f28bf7fe640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c0163d80 0x7f28c015ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:42.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.756+0000 7f28beffd640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c015a500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28b000efc0 tx=0x7f28b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:42.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.756+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0018020 con 0x7f28c00d0020 2024-04-25T15:19:42.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.756+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b0009e40 con 0x7f28c00d0020 2024-04-25T15:19:42.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.756+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c01ea870 con 0x7f28c00d0020 2024-04-25T15:19:42.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.756+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b00146a0 con 0x7f28c00d0020 2024-04-25T15:19:42.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.756+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c01eae30 con 0x7f28c00d0020 2024-04-25T15:19:42.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.757+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c006a0e0 con 0x7f28c00d0020 2024-04-25T15:19:42.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.757+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f28b0005d10 con 0x7f28c00d0020 2024-04-25T15:19:42.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.758+0000 7f28a7fff640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28ac0744b0 0x7f28ac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:42.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.758+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f28b0095bc0 con 0x7f28c00d0020 2024-04-25T15:19:42.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.758+0000 7f28be7fc640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28ac0744b0 0x7f28ac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:42.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.758+0000 7f28be7fc640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28ac0744b0 0x7f28ac0769c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f28b4034000 tx=0x7f28b402c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:42.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.763+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f28b0022030 con 0x7f28c00d0020 2024-04-25T15:19:42.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.976+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28c00d4100 con 0x7f28c00d0020 2024-04-25T15:19:42.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.976+0000 7f28a7fff640 1 -- 172.21.15.146:0/2253804449 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3876 (secure 0 0 0) 0x7f28b0010040 con 0x7f28c00d0020 2024-04-25T15:19:42.977 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:42.977 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T15:19:42.977 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 3 2024-04-25T15:19:42.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28ac0744b0 msgr2=0x7f28ac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:42.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28ac0744b0 0x7f28ac0769c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f28b4034000 tx=0x7f28b402c000 comp rx=0 tx=0).stop 2024-04-25T15:19:42.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 msgr2=0x7f28c015a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:42.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c015a500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28b000efc0 tx=0x7f28b000c490 comp rx=0 tx=0).stop 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 shutdown_connections 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28ac0744b0 0x7f28ac0769c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28c0163d80 0x7f28c015ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28c00d4f60 0x7f28c015d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 --2- 172.21.15.146:0/2253804449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28c00d0020 0x7f28c015a500 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 >> 172.21.15.146:0/2253804449 conn(0x7f28c00d2450 msgr2=0x7f28c00b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 shutdown_connections 2024-04-25T15:19:42.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:42.978+0000 7f28c59ce640 1 -- 172.21.15.146:0/2253804449 wait complete. 2024-04-25T15:19:42.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:43.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.098+0000 7f742e1d7640 1 Processor -- start 2024-04-25T15:19:43.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.098+0000 7f742e1d7640 1 -- start start 2024-04-25T15:19:43.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f742e1d7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f742815ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f742e1d7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f74280b45d0 0x7f74280d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f742e1d7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74280d1050 con 0x7f74280d0c40 2024-04-25T15:19:43.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f742e1d7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f74280d4db0 con 0x7f74280b45d0 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f742e1d7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f742806de00 con 0x7f74280d12a0 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f7426ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f74280b45d0 0x7f74280d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f74277fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f742815ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f74277fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f742815ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:51878/0 (socket says 172.21.15.146:51878) 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f7427fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f74280d0c40 0x7f7428163b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:46898/0 (socket says 172.21.15.146:46898) 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f7426ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f74280b45d0 0x7f74280d0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:43272/0 (socket says 172.21.15.146:43272) 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.099+0000 7f74277fe640 1 -- 172.21.15.146:0/3162983688 learned_addr learned my addr 172.21.15.146:0/3162983688 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.100+0000 7f74277fe640 1 -- 172.21.15.146:0/3162983688 >> v2:172.21.15.177:3300/0 conn(0x7f74280b45d0 msgr2=0x7f74280d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.100+0000 7f74277fe640 1 --2- 172.21.15.146:0/3162983688 >> v2:172.21.15.177:3300/0 conn(0x7f74280b45d0 0x7f74280d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.100+0000 7f74277fe640 1 -- 172.21.15.146:0/3162983688 >> v1:172.21.15.177:6789/0 conn(0x7f74280d0c40 legacy=0x7f7428163b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.100+0000 7f74277fe640 1 -- 172.21.15.146:0/3162983688 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74280d5000 con 0x7f74280d12a0 2024-04-25T15:19:43.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.100+0000 7f74277fe640 1 --2- 172.21.15.146:0/3162983688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f742815ff70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7418009870 tx=0x7f741802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ebb6e6a8bb0762c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f74267fc640 1 -- 172.21.15.146:0/3162983688 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418037070 con 0x7f74280d12a0 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f74267fc640 1 -- 172.21.15.146:0/3162983688 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7418032440 con 0x7f74280d12a0 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f74267fc640 1 -- 172.21.15.146:0/3162983688 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418032810 con 0x7f74280d12a0 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 -- 172.21.15.146:0/3162983688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 msgr2=0x7f742815ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 --2- 172.21.15.146:0/3162983688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f742815ff70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7418009870 tx=0x7f741802a130 comp rx=0 tx=0).stop 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 -- 172.21.15.146:0/3162983688 shutdown_connections 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 --2- 172.21.15.146:0/3162983688 >> v2:172.21.15.177:3300/0 conn(0x7f74280b45d0 0x7f74280d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 --2- 172.21.15.146:0/3162983688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f742815ff70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 -- 172.21.15.146:0/3162983688 >> 172.21.15.146:0/3162983688 conn(0x7f74280d63b0 msgr2=0x7f74280d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 -- 172.21.15.146:0/3162983688 shutdown_connections 2024-04-25T15:19:43.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.101+0000 7f742e1d7640 1 -- 172.21.15.146:0/3162983688 wait complete. 2024-04-25T15:19:43.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.103+0000 7f742e1d7640 1 Processor -- start 2024-04-25T15:19:43.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.103+0000 7f742e1d7640 1 -- start start 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.104+0000 7f742e1d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74280b45d0 0x7f742815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.104+0000 7f742e1d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 0x7f74280ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.104+0000 7f742e1d7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f74280cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f742e1d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f742806c8d0 con 0x7f74280d0c40 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f742e1d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f742806c5f0 con 0x7f74280b45d0 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f742e1d7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f74280b8f10 con 0x7f74280d12a0 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 0x7f74280ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 0x7f74280ce950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36060/0 (socket says 172.21.15.146:36060) 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 -- 172.21.15.146:0/2389765401 learned_addr learned my addr 172.21.15.146:0/2389765401 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f74277fe640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74280b45d0 0x7f742815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7427fff640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f74280cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 -- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 msgr2=0x7f74280cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f74280cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 -- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74280b45d0 msgr2=0x7f742815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74280b45d0 0x7f742815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 -- 172.21.15.146:0/2389765401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74280ccbf0 con 0x7f74280d0c40 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f7426ffd640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 0x7f74280ce950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f741c00e840 tx=0x7f741c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c017070 con 0x7f74280d0c40 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f741c00c940 con 0x7f74280d0c40 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74280cced0 con 0x7f74280d0c40 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c00cd10 con 0x7f74280d0c40 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.105+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74280cd490 con 0x7f74280d0c40 2024-04-25T15:19:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.106+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742806a0e0 con 0x7f74280d0c40 2024-04-25T15:19:43.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.107+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f741c01a070 con 0x7f74280d0c40 2024-04-25T15:19:43.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.107+0000 7f740ffff640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f74140744b0 0x7f74140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.108+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f741c095040 con 0x7f74280d0c40 2024-04-25T15:19:43.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.108+0000 7f74277fe640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f74140744b0 0x7f74140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.108+0000 7f74277fe640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f74140744b0 0x7f74140769c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7418034000 tx=0x7f741802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.112+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f741c05f560 con 0x7f74280d0c40 2024-04-25T15:19:43.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.288+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74280d3e30 con 0x7f74280d0c40 2024-04-25T15:19:43.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.289+0000 7f740ffff640 1 -- 172.21.15.146:0/2389765401 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12932 (secure 0 0 0) 0x7f741c063860 con 0x7f74280d0c40 2024-04-25T15:19:43.289 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:43.290 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":16,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:36.044209+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f74140744b0 msgr2=0x7f74140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f74140744b0 0x7f74140769c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7418034000 tx=0x7f741802e000 comp rx=0 tx=0).stop 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 msgr2=0x7f74280ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 0x7f74280ce950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f741c00e840 tx=0x7f741c00ed00 comp rx=0 tx=0).stop 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 shutdown_connections 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f74140744b0 0x7f74140769c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74280d12a0 0x7f74280cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74280d0c40 0x7f74280ce950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 --2- 172.21.15.146:0/2389765401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74280b45d0 0x7f742815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 >> 172.21.15.146:0/2389765401 conn(0x7f74280d63b0 msgr2=0x7f74280b5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 shutdown_connections 2024-04-25T15:19:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.291+0000 7f742e1d7640 1 -- 172.21.15.146:0/2389765401 wait complete. 2024-04-25T15:19:43.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:19:43.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:19:43.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.413+0000 7f7a58d35640 1 Processor -- start 2024-04-25T15:19:43.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.413+0000 7f7a58d35640 1 -- start start 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a58d35640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a541602e0 0x7f7a540b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a58d35640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a5409b5e0 0x7f7a5409b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a58d35640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a540cf130 0x7f7a540cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a58d35640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f7a541606f0 con 0x7f7a540cf130 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a58d35640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f7a5409bf30 con 0x7f7a5409b5e0 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a58d35640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7a540d2480 con 0x7f7a541602e0 2024-04-25T15:19:43.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a52d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a540cf130 0x7f7a540cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a52d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a540cf130 0x7f7a540cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36070/0 (socket says 172.21.15.146:36070) 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a52d9d640 1 -- 172.21.15.146:0/2750165453 learned_addr learned my addr 172.21.15.146:0/2750165453 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.414+0000 7f7a5259c640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a541602e0 0x7f7a540b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a52d9d640 1 -- 172.21.15.146:0/2750165453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a541602e0 msgr2=0x7f7a540b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a51d9b640 1 --2- 172.21.15.146:0/2750165453 >> v2:172.21.15.177:3300/0 conn(0x7f7a5409b5e0 0x7f7a5409b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a52d9d640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a541602e0 0x7f7a540b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a52d9d640 1 -- 172.21.15.146:0/2750165453 >> v2:172.21.15.177:3300/0 conn(0x7f7a5409b5e0 msgr2=0x7f7a5409b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a52d9d640 1 --2- 172.21.15.146:0/2750165453 >> v2:172.21.15.177:3300/0 conn(0x7f7a5409b5e0 0x7f7a5409b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a52d9d640 1 -- 172.21.15.146:0/2750165453 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a540d2740 con 0x7f7a540cf130 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a5259c640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a541602e0 0x7f7a540b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a52d9d640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a540cf130 0x7f7a540cf540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a540d0340 tx=0x7f7a4c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7e7d286a9d3e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a5159a640 1 -- 172.21.15.146:0/2750165453 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c037070 con 0x7f7a540cf130 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a5159a640 1 -- 172.21.15.146:0/2750165453 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a4c032440 con 0x7f7a540cf130 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.415+0000 7f7a5159a640 1 -- 172.21.15.146:0/2750165453 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c032810 con 0x7f7a540cf130 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 -- 172.21.15.146:0/2750165453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a540cf130 msgr2=0x7f7a540cf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a540cf130 0x7f7a540cf540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a540d0340 tx=0x7f7a4c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 -- 172.21.15.146:0/2750165453 shutdown_connections 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a540cf130 0x7f7a540cf540 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2750165453 >> v2:172.21.15.177:3300/0 conn(0x7f7a5409b5e0 0x7f7a5409b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2750165453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a541602e0 0x7f7a540b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 -- 172.21.15.146:0/2750165453 >> 172.21.15.146:0/2750165453 conn(0x7f7a540b7490 msgr2=0x7f7a540b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 -- 172.21.15.146:0/2750165453 shutdown_connections 2024-04-25T15:19:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.416+0000 7f7a58d35640 1 -- 172.21.15.146:0/2750165453 wait complete. 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.418+0000 7f7a58d35640 1 Processor -- start 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.418+0000 7f7a58d35640 1 -- start start 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.419+0000 7f7a58d35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 0x7f7a54163190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.419+0000 7f7a58d35640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a540cf130 0x7f7a5415b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.419+0000 7f7a58d35640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a541602e0 0x7f7a540c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.419+0000 7f7a58d35640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7a540d2fe0 con 0x7f7a5409b5e0 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.419+0000 7f7a58d35640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a540d2d00 con 0x7f7a541602e0 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.419+0000 7f7a58d35640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7a5406f7f0 con 0x7f7a540cf130 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 0x7f7a54163190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 0x7f7a54163190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36076/0 (socket says 172.21.15.146:36076) 2024-04-25T15:19:43.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 -- 172.21.15.146:0/2392359824 learned_addr learned my addr 172.21.15.146:0/2392359824 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:43.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a52d9d640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a541602e0 0x7f7a540c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a51d9b640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a540cf130 0x7f7a5415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 -- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a540cf130 msgr2=0x7f7a5415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a540cf130 0x7f7a5415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 -- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a541602e0 msgr2=0x7f7a540c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a541602e0 0x7f7a540c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 -- 172.21.15.146:0/2392359824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5415b970 con 0x7f7a5409b5e0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a52d9d640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a541602e0 0x7f7a540c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a5259c640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 0x7f7a54163190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400cde0 tx=0x7f7a4400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4400f680 con 0x7f7a5409b5e0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a4400f820 con 0x7f7a5409b5e0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a540b2690 con 0x7f7a5409b5e0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4400fc10 con 0x7f7a5409b5e0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.420+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a540b2c50 con 0x7f7a5409b5e0 2024-04-25T15:19:43.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.421+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a5406a0e0 con 0x7f7a5409b5e0 2024-04-25T15:19:43.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.422+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7a44018400 con 0x7f7a5409b5e0 2024-04-25T15:19:43.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.422+0000 7f7a42ffd640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a340744e0 0x7f7a340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.422+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f7a44095c80 con 0x7f7a5409b5e0 2024-04-25T15:19:43.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.422+0000 7f7a51d9b640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a340744e0 0x7f7a340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.423+0000 7f7a51d9b640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a340744e0 0x7f7a340769f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7a540b4990 tx=0x7f7a48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.427+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a44014030 con 0x7f7a5409b5e0 2024-04-25T15:19:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.602+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7a540ccd90 con 0x7f7a5409b5e0 2024-04-25T15:19:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.603+0000 7f7a42ffd640 1 -- 172.21.15.146:0/2392359824 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f7a44060760 con 0x7f7a5409b5e0 2024-04-25T15:19:43.604 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:19:43.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a340744e0 msgr2=0x7f7a340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a340744e0 0x7f7a340769f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7a540b4990 tx=0x7f7a48008040 comp rx=0 tx=0).stop 2024-04-25T15:19:43.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 msgr2=0x7f7a54163190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 0x7f7a54163190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400cde0 tx=0x7f7a4400a650 comp rx=0 tx=0).stop 2024-04-25T15:19:43.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 shutdown_connections 2024-04-25T15:19:43.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a340744e0 0x7f7a340769f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a541602e0 0x7f7a540c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a540cf130 0x7f7a5415b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 --2- 172.21.15.146:0/2392359824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a5409b5e0 0x7f7a54163190 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 >> 172.21.15.146:0/2392359824 conn(0x7f7a540b7490 msgr2=0x7f7a5406edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 shutdown_connections 2024-04-25T15:19:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.605+0000 7f7a58d35640 1 -- 172.21.15.146:0/2392359824 wait complete. 2024-04-25T15:19:43.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:19:43.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.724+0000 7f5e2b892640 1 Processor -- start 2024-04-25T15:19:43.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.725+0000 7f5e2b892640 1 -- start start 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2b892640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5e240d12a0 0x7f5e2415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2b892640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e240d0e10 0x7f5e2406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2b892640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e240d5000 con 0x7f5e240b45d0 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2b892640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5e240d4db0 con 0x7f5e240d0e10 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2b892640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5e240d52e0 con 0x7f5e240d12a0 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e240d0e10 0x7f5e2406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e240d0e10 0x7f5e2406cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36078/0 (socket says 172.21.15.146:36078) 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 -- 172.21.15.146:0/2578729167 learned_addr learned my addr 172.21.15.146:0/2578729167 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e28e2d640 1 --1- 172.21.15.146:0/2578729167 >> v1:172.21.15.146:6789/0 conn(0x7f5e240b45d0 0x7f5e240d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:57872/0 (socket says 172.21.15.146:57872) 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2962e640 1 --2- 172.21.15.146:0/2578729167 >> v2:172.21.15.177:3300/0 conn(0x7f5e240d12a0 0x7f5e2415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 -- 172.21.15.146:0/2578729167 >> v2:172.21.15.177:3300/0 conn(0x7f5e240d12a0 msgr2=0x7f5e2415ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 --2- 172.21.15.146:0/2578729167 >> v2:172.21.15.177:3300/0 conn(0x7f5e240d12a0 0x7f5e2415ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 -- 172.21.15.146:0/2578729167 >> v1:172.21.15.146:6789/0 conn(0x7f5e240b45d0 legacy=0x7f5e240d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e29e2f640 1 -- 172.21.15.146:0/2578729167 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2406c5f0 con 0x7f5e240d0e10 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e1bfff640 1 -- 172.21.15.146:0/2578729167 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167106014 0 0) 0x7f5e240d5000 con 0x7f5e240b45d0 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.726+0000 7f5e2962e640 1 --2- 172.21.15.146:0/2578729167 >> v2:172.21.15.177:3300/0 conn(0x7f5e240d12a0 0x7f5e2415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.727+0000 7f5e29e2f640 1 --2- 172.21.15.146:0/2578729167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d0e10 0x7f5e2406cdf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e240d3720 tx=0x7f5e1c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c24f1d5ea6fbe90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.727+0000 7f5e1bfff640 1 -- 172.21.15.146:0/2578729167 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c036070 con 0x7f5e240d0e10 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.727+0000 7f5e1bfff640 1 -- 172.21.15.146:0/2578729167 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1c031600 con 0x7f5e240d0e10 2024-04-25T15:19:43.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.727+0000 7f5e1bfff640 1 -- 172.21.15.146:0/2578729167 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0319d0 con 0x7f5e240d0e10 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 -- 172.21.15.146:0/2578729167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d0e10 msgr2=0x7f5e2406cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 --2- 172.21.15.146:0/2578729167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d0e10 0x7f5e2406cdf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e240d3720 tx=0x7f5e1c02a610 comp rx=0 tx=0).stop 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 -- 172.21.15.146:0/2578729167 shutdown_connections 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 --2- 172.21.15.146:0/2578729167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d0e10 0x7f5e2406cdf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 --2- 172.21.15.146:0/2578729167 >> v2:172.21.15.177:3300/0 conn(0x7f5e240d12a0 0x7f5e2415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 -- 172.21.15.146:0/2578729167 >> 172.21.15.146:0/2578729167 conn(0x7f5e240d63b0 msgr2=0x7f5e240d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 -- 172.21.15.146:0/2578729167 shutdown_connections 2024-04-25T15:19:43.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.728+0000 7f5e2b892640 1 -- 172.21.15.146:0/2578729167 wait complete. 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.730+0000 7f5e2b892640 1 Processor -- start 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.730+0000 7f5e2b892640 1 -- start start 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2b892640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 0x7f5e24153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2b892640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e240d12a0 0x7f5e24160b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2b892640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24155340 0x7f5e24159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2b892640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240b9090 con 0x7f5e240b45d0 2024-04-25T15:19:43.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2b892640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240b8f10 con 0x7f5e24155340 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2b892640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5e240b4b20 con 0x7f5e240d12a0 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 0x7f5e24153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 0x7f5e24153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36086/0 (socket says 172.21.15.146:36086) 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 -- 172.21.15.146:0/1708382067 learned_addr learned my addr 172.21.15.146:0/1708382067 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 -- 172.21.15.146:0/1708382067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e240d12a0 msgr2=0x7f5e24160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e240d12a0 0x7f5e24160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 -- 172.21.15.146:0/1708382067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24155340 msgr2=0x7f5e24159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24155340 0x7f5e24159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.731+0000 7f5e2962e640 1 -- 172.21.15.146:0/1708382067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e24155ca0 con 0x7f5e240b45d0 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.732+0000 7f5e2962e640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 0x7f5e24153960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400c7f0 tx=0x7f5e1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.732+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14015070 con 0x7f5e240b45d0 2024-04-25T15:19:43.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.732+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e14007820 con 0x7f5e240b45d0 2024-04-25T15:19:43.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.732+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e24162b60 con 0x7f5e240b45d0 2024-04-25T15:19:43.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.732+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14007c10 con 0x7f5e240b45d0 2024-04-25T15:19:43.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.732+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e24163120 con 0x7f5e240b45d0 2024-04-25T15:19:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.733+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2406a0e0 con 0x7f5e240b45d0 2024-04-25T15:19:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.734+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5e14018070 con 0x7f5e240b45d0 2024-04-25T15:19:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.734+0000 7f5e19ffb640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e080744e0 0x7f5e080769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.734+0000 7f5e28e2d640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e080744e0 0x7f5e080769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.734+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f5e14094ca0 con 0x7f5e240b45d0 2024-04-25T15:19:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.735+0000 7f5e28e2d640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e080744e0 0x7f5e080769f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e20007fd0 tx=0x7f5e20007960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:43.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.739+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e1405f780 con 0x7f5e240b45d0 2024-04-25T15:19:43.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.917+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5e2415d660 con 0x7f5e240b45d0 2024-04-25T15:19:43.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.917+0000 7f5e19ffb640 1 -- 172.21.15.146:0/1708382067 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f5e1405f080 con 0x7f5e240b45d0 2024-04-25T15:19:43.945 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:43.946 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2024-04-25T15:19:43.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:19:43.946 INFO:teuthology.orchestra.run.smithi146.stdout: "poolnum": 1, 2024-04-25T15:19:43.946 INFO:teuthology.orchestra.run.smithi146.stdout: "poolname": ".mgr" 2024-04-25T15:19:43.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:19:43.946 INFO:teuthology.orchestra.run.smithi146.stdout:] 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e080744e0 msgr2=0x7f5e080769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e080744e0 0x7f5e080769f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e20007fd0 tx=0x7f5e20007960 comp rx=0 tx=0).stop 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 msgr2=0x7f5e24153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 0x7f5e24153960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400c7f0 tx=0x7f5e1400ccb0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 shutdown_connections 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e080744e0 0x7f5e080769f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e24155340 0x7f5e24159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e240d12a0 0x7f5e24160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 --2- 172.21.15.146:0/1708382067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240b45d0 0x7f5e24153960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 >> 172.21.15.146:0/1708382067 conn(0x7f5e240d63b0 msgr2=0x7f5e240baba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:43.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 shutdown_connections 2024-04-25T15:19:43.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:43.919+0000 7f5e2b892640 1 -- 172.21.15.146:0/1708382067 wait complete. 2024-04-25T15:19:43.949 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:19:43.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:19:44.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.060+0000 7fb7182d8640 1 Processor -- start 2024-04-25T15:19:44.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.060+0000 7fb7182d8640 1 -- start start 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb7182d8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7100d4d00 0x7fb7100b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb7182d8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb710160de0 con 0x7fb71006cbf0 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb7182d8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7101610c0 con 0x7fb7100d0780 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb7182d8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb7100d5450 con 0x7fb7100d4d00 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb716875640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb71006cbf0 0x7fb71006d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:57880/0 (socket says 172.21.15.146:57880) 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb716875640 1 -- 172.21.15.146:0/1462322073 learned_addr learned my addr 172.21.15.146:0/1462322073 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:44.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb715873640 1 --2- 172.21.15.146:0/1462322073 >> v2:172.21.15.177:3300/0 conn(0x7fb7100d4d00 0x7fb7100b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb715072640 1 -- 172.21.15.146:0/1462322073 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216156495 0 0) 0x7fb710160de0 con 0x7fb71006cbf0 2024-04-25T15:19:44.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.062+0000 7fb715072640 1 -- 172.21.15.146:0/1462322073 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6fc003520 con 0x7fb71006cbf0 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.063+0000 7fb715873640 1 -- 172.21.15.146:0/1462322073 >> v1:172.21.15.177:6789/0 conn(0x7fb7100d0780 legacy=0x7fb7100d45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.063+0000 7fb715873640 1 -- 172.21.15.146:0/1462322073 >> v1:172.21.15.146:6789/0 conn(0x7fb71006cbf0 legacy=0x7fb71006d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.063+0000 7fb715072640 1 -- 172.21.15.146:0/1462322073 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3633983128 0 0) 0x7fb7101610c0 con 0x7fb7100d0780 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.063+0000 7fb715873640 1 -- 172.21.15.146:0/1462322073 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7101613a0 con 0x7fb7100d4d00 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.063+0000 7fb715873640 1 --2- 172.21.15.146:0/1462322073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7100d4d00 0x7fb7100b4ec0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7000097a0 tx=0x7fb70002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cea56a8ae15951a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.063+0000 7fb715072640 1 -- 172.21.15.146:0/1462322073 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700037070 con 0x7fb7100d4d00 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb715072640 1 -- 172.21.15.146:0/1462322073 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb700032440 con 0x7fb7100d4d00 2024-04-25T15:19:44.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb715072640 1 -- 172.21.15.146:0/1462322073 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700032810 con 0x7fb7100d4d00 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 -- 172.21.15.146:0/1462322073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7100d4d00 msgr2=0x7fb7100b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1462322073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7100d4d00 0x7fb7100b4ec0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7000097a0 tx=0x7fb70002a230 comp rx=0 tx=0).stop 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 -- 172.21.15.146:0/1462322073 shutdown_connections 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1462322073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7100d4d00 0x7fb7100b4ec0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 -- 172.21.15.146:0/1462322073 >> 172.21.15.146:0/1462322073 conn(0x7fb7100d2b90 msgr2=0x7fb7100d59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 -- 172.21.15.146:0/1462322073 shutdown_connections 2024-04-25T15:19:44.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.064+0000 7fb7182d8640 1 -- 172.21.15.146:0/1462322073 wait complete. 2024-04-25T15:19:44.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.066+0000 7fb7182d8640 1 Processor -- start 2024-04-25T15:19:44.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.066+0000 7fb7182d8640 1 -- start start 2024-04-25T15:19:44.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.067+0000 7fb7182d8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 0x7fb7100cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.067+0000 7fb7182d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7100d0780 0x7fb710149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.067+0000 7fb7182d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb71014a780 0x7fb71014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.067+0000 7fb7182d8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb710160a00 con 0x7fb71006cbf0 2024-04-25T15:19:44.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.067+0000 7fb7182d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb710160880 con 0x7fb71014a780 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.067+0000 7fb7182d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb710160b80 con 0x7fb7100d0780 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 0x7fb7100cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 0x7fb7100cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36094/0 (socket says 172.21.15.146:36094) 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 -- 172.21.15.146:0/1311818853 learned_addr learned my addr 172.21.15.146:0/1311818853 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716875640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb71014a780 0x7fb71014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb715873640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7100d0780 0x7fb710149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 -- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7100d0780 msgr2=0x7fb710149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7100d0780 0x7fb710149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 -- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb71014a780 msgr2=0x7fb71014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb71014a780 0x7fb71014a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 -- 172.21.15.146:0/1311818853 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7100bab20 con 0x7fb71006cbf0 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716875640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb71014a780 0x7fb71014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:44.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb716074640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 0x7fb7100cf5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c760 tx=0x7fb70c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:44.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c017070 con 0x7fb71006cbf0 2024-04-25T15:19:44.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7100ba840 con 0x7fb71006cbf0 2024-04-25T15:19:44.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70c003da0 con 0x7fb71006cbf0 2024-04-25T15:19:44.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c005290 con 0x7fb71006cbf0 2024-04-25T15:19:44.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.068+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb710147e20 con 0x7fb71006cbf0 2024-04-25T15:19:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.069+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb71006a0e0 con 0x7fb71006cbf0 2024-04-25T15:19:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.070+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb70c01a070 con 0x7fb71006cbf0 2024-04-25T15:19:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.070+0000 7fb706ffd640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb6f40744b0 0x7fb6f40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.071+0000 7fb715873640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb6f40744b0 0x7fb6f40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.071+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fb70c0944c0 con 0x7fb71006cbf0 2024-04-25T15:19:44.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.071+0000 7fb715873640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb6f40744b0 0x7fb6f40769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb700034000 tx=0x7fb70002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:44.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.075+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb70c05e960 con 0x7fb71006cbf0 2024-04-25T15:19:44.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.250+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb7100d3e80 con 0x7fb71006cbf0 2024-04-25T15:19:44.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.435+0000 7fb706ffd640 1 -- 172.21.15.146:0/1311818853 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fb70c062c60 con 0x7fb71006cbf0 2024-04-25T15:19:44.436 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' created 2024-04-25T15:19:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb6f40744b0 msgr2=0x7fb6f40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb6f40744b0 0x7fb6f40769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb700034000 tx=0x7fb70002e000 comp rx=0 tx=0).stop 2024-04-25T15:19:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 msgr2=0x7fb7100cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 0x7fb7100cf5e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c760 tx=0x7fb70c00cc20 comp rx=0 tx=0).stop 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 shutdown_connections 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb6f40744b0 0x7fb6f40769c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb71014a780 0x7fb71014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7100d0780 0x7fb710149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 --2- 172.21.15.146:0/1311818853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb71006cbf0 0x7fb7100cf5e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 >> 172.21.15.146:0/1311818853 conn(0x7fb7100d2b90 msgr2=0x7fb71015fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 shutdown_connections 2024-04-25T15:19:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.437+0000 7fb7182d8640 1 -- 172.21.15.146:0/1311818853 wait complete. 2024-04-25T15:19:44.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T15:19:44.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.565+0000 7fc28dd38640 1 Processor -- start 2024-04-25T15:19:44.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.565+0000 7fc28dd38640 1 -- start start 2024-04-25T15:19:44.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.566+0000 7fc28dd38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880b4ad0 0x7fc28815ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.566+0000 7fc28dd38640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2880d0730 0x7fc2880d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.566+0000 7fc28dd38640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28806c5f0 con 0x7fc2880d4de0 2024-04-25T15:19:44.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.566+0000 7fc28dd38640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc28806ceb0 con 0x7fc2880d0730 2024-04-25T15:19:44.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.566+0000 7fc28dd38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc2880d1080 con 0x7fc2880b4ad0 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.566+0000 7fc287fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2880d0730 0x7fc2880d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc287fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2880d0730 0x7fc2880d0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36106/0 (socket says 172.21.15.146:36106) 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc287fff640 1 -- 172.21.15.146:0/2510654853 learned_addr learned my addr 172.21.15.146:0/2510654853 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc286ffd640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880b4ad0 0x7fc28815ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc287fff640 1 -- 172.21.15.146:0/2510654853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880b4ad0 msgr2=0x7fc28815ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc287fff640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880b4ad0 0x7fc28815ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc287fff640 1 -- 172.21.15.146:0/2510654853 >> v1:172.21.15.177:6789/0 conn(0x7fc2880d4de0 legacy=0x7fc2880b8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc287fff640 1 -- 172.21.15.146:0/2510654853 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28806c860 con 0x7fc2880d0730 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.567+0000 7fc286ffd640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880b4ad0 0x7fc28815ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc287fff640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d0730 0x7fc2880d0b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc280009e50 tx=0x7fc28002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d627a2fa2ffaabc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc2867fc640 1 -- 172.21.15.146:0/2510654853 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc280036070 con 0x7fc2880d0730 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc2867fc640 1 -- 172.21.15.146:0/2510654853 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc280031600 con 0x7fc2880d0730 2024-04-25T15:19:44.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc2867fc640 1 -- 172.21.15.146:0/2510654853 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2800319d0 con 0x7fc2880d0730 2024-04-25T15:19:44.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 -- 172.21.15.146:0/2510654853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d0730 msgr2=0x7fc2880d0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d0730 0x7fc2880d0b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc280009e50 tx=0x7fc28002a680 comp rx=0 tx=0).stop 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 -- 172.21.15.146:0/2510654853 shutdown_connections 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d0730 0x7fc2880d0b40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 --2- 172.21.15.146:0/2510654853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880b4ad0 0x7fc28815ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 -- 172.21.15.146:0/2510654853 >> 172.21.15.146:0/2510654853 conn(0x7fc2880d6240 msgr2=0x7fc2880b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 -- 172.21.15.146:0/2510654853 shutdown_connections 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.568+0000 7fc28dd38640 1 -- 172.21.15.146:0/2510654853 wait complete. 2024-04-25T15:19:44.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.570+0000 7fc28dd38640 1 Processor -- start 2024-04-25T15:19:44.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.570+0000 7fc28dd38640 1 -- start start 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc28dd38640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2880b4ad0 0x7fc288153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc28dd38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880d0730 0x7fc288158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc28dd38640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 0x7fc288157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc28dd38640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2880d1380 con 0x7fc2880d4de0 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc28dd38640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc2880d5330 con 0x7fc2880b4ad0 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc28dd38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc2880d1500 con 0x7fc2880d0730 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 0x7fc288157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 0x7fc288157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36122/0 (socket says 172.21.15.146:36122) 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 -- 172.21.15.146:0/293683981 learned_addr learned my addr 172.21.15.146:0/293683981 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc2877fe640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2880b4ad0 0x7fc288153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc286ffd640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880d0730 0x7fc288158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 -- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880d0730 msgr2=0x7fc288158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880d0730 0x7fc288158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 -- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2880b4ad0 msgr2=0x7fc288153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2880b4ad0 0x7fc288153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 -- 172.21.15.146:0/293683981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28815b570 con 0x7fc2880d4de0 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc287fff640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 0x7fc288157070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc280034040 tx=0x7fc280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.572+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28003f020 con 0x7fc2880d4de0 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.573+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc280037040 con 0x7fc2880d4de0 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.573+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc288159af0 con 0x7fc2880d4de0 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.573+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc280036070 con 0x7fc2880d4de0 2024-04-25T15:19:44.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.573+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc288154fe0 con 0x7fc2880d4de0 2024-04-25T15:19:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.574+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2880507b0 con 0x7fc2880d4de0 2024-04-25T15:19:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.574+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc280042020 con 0x7fc2880d4de0 2024-04-25T15:19:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.575+0000 7fc26ffff640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2740744e0 0x7fc2740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.575+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7fc2800b53d0 con 0x7fc2880d4de0 2024-04-25T15:19:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.575+0000 7fc2877fe640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2740744e0 0x7fc2740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:44.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.575+0000 7fc2877fe640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2740744e0 0x7fc2740769f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2780016e0 tx=0x7fc278009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:44.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.579+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc280049030 con 0x7fc2880d4de0 2024-04-25T15:19:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:44.757+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fc288159880 con 0x7fc2880d4de0 2024-04-25T15:19:45.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.445+0000 7fc26ffff640 1 -- 172.21.15.146:0/293683981 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7fc28007f780 con 0x7fc2880d4de0 2024-04-25T15:19:45.446 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' created 2024-04-25T15:19:45.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.448+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2740744e0 msgr2=0x7fc2740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.448+0000 7fc28dd38640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2740744e0 0x7fc2740769f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2780016e0 tx=0x7fc278009040 comp rx=0 tx=0).stop 2024-04-25T15:19:45.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.448+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 msgr2=0x7fc288157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.448+0000 7fc28dd38640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 0x7fc288157070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc280034040 tx=0x7fc280004a40 comp rx=0 tx=0).stop 2024-04-25T15:19:45.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.450+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 shutdown_connections 2024-04-25T15:19:45.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.450+0000 7fc28dd38640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2740744e0 0x7fc2740769f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.450+0000 7fc28dd38640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2880d4de0 0x7fc288157070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.450+0000 7fc28dd38640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2880d0730 0x7fc288158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.450+0000 7fc28dd38640 1 --2- 172.21.15.146:0/293683981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2880b4ad0 0x7fc288153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.450+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 >> 172.21.15.146:0/293683981 conn(0x7fc2880d6240 msgr2=0x7fc28815d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.453+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 shutdown_connections 2024-04-25T15:19:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.454+0000 7fc28dd38640 1 -- 172.21.15.146:0/293683981 wait complete. 2024-04-25T15:19:45.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:19:45.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.593+0000 7f82a2242640 1 Processor -- start 2024-04-25T15:19:45.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.593+0000 7f82a2242640 1 -- start start 2024-04-25T15:19:45.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.594+0000 7f82a2242640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c09b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.594+0000 7f82a2242640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f829c09bf30 con 0x7f829c1602e0 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.594+0000 7f82a2242640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f829c0d2480 con 0x7f829c0cd290 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.594+0000 7f82a2242640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f829c1606f0 con 0x7f829c09b5e0 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f829c1602e0 0x7f829c0b18c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:57894/0 (socket says 172.21.15.146:57894) 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829b7fe640 1 -- 172.21.15.146:0/2768028081 learned_addr learned my addr 172.21.15.146:0/2768028081 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829affd640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c09b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421584106 0 0) 0x7f828c001110 con 0x7f829c1602e0 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8284003520 con 0x7f829c1602e0 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226043839 0 0) 0x7f829c0d2480 con 0x7f829c0cd290 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f828c001110 con 0x7f829c0cd290 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1238117696 0 0) 0x7f8284003520 con 0x7f829c1602e0 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 msgr2=0x7f829c09b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:45.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c09b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 >> v1:172.21.15.177:6789/0 conn(0x7f829c0cd290 legacy=0x7f829c0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c0d2740 con 0x7f829c1602e0 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f828c001840 con 0x7f829c1602e0 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 >> v1:172.21.15.146:6789/0 conn(0x7f829c1602e0 legacy=0x7f829c0b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f8284003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8284004120 0x7f8284004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8284004d40 0x7f8284005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1606f0 con 0x7f8284004d40 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f828400bb80 con 0x7f8284004120 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f828400bd80 con 0x7f829c09b5e0 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829bfff640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8284004d40 0x7f8284005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829b7fe640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f8284003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.595+0000 7f829affd640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8284004120 0x7f8284004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829b7fe640 1 -- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8284004120 msgr2=0x7f8284004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:45.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829b7fe640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8284004120 0x7f8284004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829b7fe640 1 -- 172.21.15.146:0/2768028081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8284004d40 msgr2=0x7f8284005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:45.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829b7fe640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8284004d40 0x7f8284005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829b7fe640 1 -- 172.21.15.146:0/2768028081 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8284003520 con 0x7f829c09b5e0 2024-04-25T15:19:45.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829bfff640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8284004d40 0x7f8284005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:45.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.596+0000 7f829b7fe640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f8284003b70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f829c163cd0 tx=0x7f8284005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dcc5855ce4743bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:45.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c007d90 con 0x7f829c09b5e0 2024-04-25T15:19:45.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f828c005af0 con 0x7f829c09b5e0 2024-04-25T15:19:45.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f829a7fc640 1 -- 172.21.15.146:0/2768028081 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c007070 con 0x7f829c09b5e0 2024-04-25T15:19:45.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f82a2242640 1 -- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 msgr2=0x7f8284003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f82a2242640 1 --2- 172.21.15.146:0/2768028081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f8284003b70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f829c163cd0 tx=0x7f8284005830 comp rx=0 tx=0).stop 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f829bfff640 1 -- 172.21.15.146:0/2768028081 reap_dead start 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f82a2242640 1 -- 172.21.15.146:0/2768028081 shutdown_connections 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f82a2242640 1 -- 172.21.15.146:0/2768028081 >> 172.21.15.146:0/2768028081 conn(0x7f829c0b7490 msgr2=0x7f829c0b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f82a2242640 1 -- 172.21.15.146:0/2768028081 shutdown_connections 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.597+0000 7f82a2242640 1 -- 172.21.15.146:0/2768028081 wait complete. 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.599+0000 7f82a2242640 1 Processor -- start 2024-04-25T15:19:45.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.599+0000 7f82a2242640 1 -- start start 2024-04-25T15:19:45.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.600+0000 7f82a2242640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c0cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.600+0000 7f82a2242640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 0x7f829c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.600+0000 7f82a2242640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c153f30 0x7f829c0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.600+0000 7f82a2242640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f829c0d2fe0 con 0x7f829c15f120 2024-04-25T15:19:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.600+0000 7f82a2242640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f829c0d2d00 con 0x7f829c153f30 2024-04-25T15:19:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.600+0000 7f82a2242640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f829c06f7f0 con 0x7f829c09b5e0 2024-04-25T15:19:45.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 0x7f829c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 0x7f829c0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36140/0 (socket says 172.21.15.146:36140) 2024-04-25T15:19:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 -- 172.21.15.146:0/1025839817 learned_addr learned my addr 172.21.15.146:0/1025839817 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829b7fe640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c0cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829bfff640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c153f30 0x7f829c0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 -- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 msgr2=0x7f829c0cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c0cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 -- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c153f30 msgr2=0x7f829c0b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c153f30 0x7f829c0b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 -- 172.21.15.146:0/1025839817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c0cacc0 con 0x7f829c15f120 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829bfff640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c153f30 0x7f829c0b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f829affd640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 0x7f829c0cf660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8290011940 tx=0x7f8290011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829000faf0 con 0x7f829c15f120 2024-04-25T15:19:45.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f829000fc90 con 0x7f829c15f120 2024-04-25T15:19:45.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c0caf80 con 0x7f829c15f120 2024-04-25T15:19:45.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290014440 con 0x7f829c15f120 2024-04-25T15:19:45.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.601+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c0cb540 con 0x7f829c15f120 2024-04-25T15:19:45.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.602+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829c0cfd40 con 0x7f829c15f120 2024-04-25T15:19:45.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.603+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8290012040 con 0x7f829c15f120 2024-04-25T15:19:45.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.604+0000 7f8283fff640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82880744e0 0x7f82880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:45.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.604+0000 7f829b7fe640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82880744e0 0x7f82880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:45.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.604+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f8290095780 con 0x7f829c15f120 2024-04-25T15:19:45.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.604+0000 7f829b7fe640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82880744e0 0x7f82880769f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f828c033000 tx=0x7f828c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:45.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.609+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8290019030 con 0x7f829c15f120 2024-04-25T15:19:45.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:19:45.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:19:45.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:45.823+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f829c06a0e0 con 0x7f829c15f120 2024-04-25T15:19:46.465 INFO:teuthology.orchestra.run.smithi146.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T15:19:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.464+0000 7f8283fff640 1 -- 172.21.15.146:0/1025839817 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f8290060090 con 0x7f829c15f120 2024-04-25T15:19:46.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.470+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82880744e0 msgr2=0x7f82880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.470+0000 7f82a2242640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82880744e0 0x7f82880769f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f828c033000 tx=0x7f828c009030 comp rx=0 tx=0).stop 2024-04-25T15:19:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.470+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 msgr2=0x7f829c0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:46.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.470+0000 7f82a2242640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 0x7f829c0cf660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8290011940 tx=0x7f8290011e00 comp rx=0 tx=0).stop 2024-04-25T15:19:46.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.472+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 shutdown_connections 2024-04-25T15:19:46.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.472+0000 7f82a2242640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82880744e0 0x7f82880769f0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f828c033000 tx=0x7f828c009030 comp rx=0 tx=0).stop 2024-04-25T15:19:46.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.472+0000 7f82a2242640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c153f30 0x7f829c0b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.472+0000 7f82a2242640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15f120 0x7f829c0cf660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.472+0000 7f82a2242640 1 --2- 172.21.15.146:0/1025839817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f829c09b5e0 0x7f829c0cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.472+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 >> 172.21.15.146:0/1025839817 conn(0x7f829c0b7490 msgr2=0x7f829c0d1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:46.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.473+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 shutdown_connections 2024-04-25T15:19:46.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.474+0000 7f82a2242640 1 -- 172.21.15.146:0/1025839817 wait complete. 2024-04-25T15:19:46.496 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T15:19:46.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T15:19:46.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.652+0000 7f1160100640 1 Processor -- start 2024-04-25T15:19:46.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.653+0000 7f1160100640 1 -- start start 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f1160100640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 0x7f115806d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f1160100640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f11580d0690 0x7f11580d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f1160100640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11580d5290 con 0x7f11580d4ad0 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f1160100640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f115806ce80 con 0x7f11580d0690 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f1160100640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f115806c550 con 0x7f115806d310 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f115de9c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f11580d4ad0 0x7f11580b8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:57908/0 (socket says 172.21.15.146:57908) 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f115de9c640 1 -- 172.21.15.146:0/1775933959 learned_addr learned my addr 172.21.15.146:0/1775933959 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f115e69d640 1 --2- 172.21.15.146:0/1775933959 >> v2:172.21.15.177:3300/0 conn(0x7f11580d0690 0x7f11580d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f115d69b640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 0x7f115806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3749155494 0 0) 0x7f11580d5290 con 0x7f11580d4ad0 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.654+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1144003520 con 0x7f11580d4ad0 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2772274454 0 0) 0x7f1144003520 con 0x7f11580d4ad0 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 msgr2=0x7f115806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 0x7f115806d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 >> v2:172.21.15.177:3300/0 conn(0x7f11580d0690 msgr2=0x7f11580d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 --2- 172.21.15.146:0/1775933959 >> v2:172.21.15.177:3300/0 conn(0x7f11580d0690 0x7f11580d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115806c830 con 0x7f11580d4ad0 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f11540019f0 con 0x7f11580d4ad0 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 >> v1:172.21.15.146:6789/0 conn(0x7f11580d4ad0 legacy=0x7f11580b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 0x7f11440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1144004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1144004d40 0x7f1144005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f115806ce80 con 0x7f1144003f30 2024-04-25T15:19:46.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f115806c550 con 0x7f1144004d40 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f114400bac0 con 0x7f115806d310 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115d69b640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1144004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115de9c640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 0x7f11440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115e69d640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1144004d40 0x7f1144005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115d69b640 1 -- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 msgr2=0x7f11440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115d69b640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f115806d310 0x7f11440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115d69b640 1 -- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1144004d40 msgr2=0x7f1144005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115d69b640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1144004d40 0x7f1144005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115d69b640 1 -- 172.21.15.146:0/1775933959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1144003520 con 0x7f1144003f30 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.655+0000 7f115e69d640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1144004d40 0x7f1144005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.656+0000 7f115d69b640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1144004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1148009a90 tx=0x7f1144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188c3ec0dca7419f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.656+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148036070 con 0x7f1144003f30 2024-04-25T15:19:46.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.656+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1148031400 con 0x7f1144003f30 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.656+0000 7f115ce9a640 1 -- 172.21.15.146:0/1775933959 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148009690 con 0x7f1144003f30 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f1160100640 1 -- 172.21.15.146:0/1775933959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 msgr2=0x7f1144004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f1160100640 1 --2- 172.21.15.146:0/1775933959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1144004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1148009a90 tx=0x7f1144005830 comp rx=0 tx=0).stop 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f115e69d640 1 -- 172.21.15.146:0/1775933959 reap_dead start 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f1160100640 1 -- 172.21.15.146:0/1775933959 shutdown_connections 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f1160100640 1 -- 172.21.15.146:0/1775933959 >> 172.21.15.146:0/1775933959 conn(0x7f11580d5a50 msgr2=0x7f11580d5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f1160100640 1 -- 172.21.15.146:0/1775933959 shutdown_connections 2024-04-25T15:19:46.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.657+0000 7f1160100640 1 -- 172.21.15.146:0/1775933959 wait complete. 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.659+0000 7f1160100640 1 Processor -- start 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.659+0000 7f1160100640 1 -- start start 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f1160100640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1158158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f1160100640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1158155880 0x7f1158143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f1160100640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1158157430 0x7f1158157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f1160100640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f11580d4ee0 con 0x7f1144003f30 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f1160100640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1158160190 con 0x7f1158157430 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f1160100640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f11581f7b00 con 0x7f1158155880 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1158158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1158158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36148/0 (socket says 172.21.15.146:36148) 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 -- 172.21.15.146:0/1666630464 learned_addr learned my addr 172.21.15.146:0/1666630464 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115e69d640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1158157430 0x7f1158157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115d69b640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1158155880 0x7f1158143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 -- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1158155880 msgr2=0x7f1158143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1158155880 0x7f1158143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 -- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1158157430 msgr2=0x7f1158157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1158157430 0x7f1158157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.660+0000 7f115de9c640 1 -- 172.21.15.146:0/1666630464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1158157d80 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.661+0000 7f115de9c640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1158158060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1154007770 tx=0x7f115400a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.661+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1154017020 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.661+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f115400ad40 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.661+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1154013630 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.661+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1158144210 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.661+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581548f0 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.662+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115806a0e0 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.662+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f115401a020 con 0x7f1144003f30 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.663+0000 7f114e7fc640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f113c0744b0 0x7f113c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:46.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.663+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f1154095ab0 con 0x7f1144003f30 2024-04-25T15:19:46.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.663+0000 7f115d69b640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f113c0744b0 0x7f113c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:46.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.663+0000 7f115d69b640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f113c0744b0 0x7f113c0769c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1148033000 tx=0x7f1148008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:46.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.668+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1154021020 con 0x7f1144003f30 2024-04-25T15:19:46.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:46.852+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f11580d3ea0 con 0x7f1144003f30 2024-04-25T15:19:47.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.467+0000 7f114e7fc640 1 -- 172.21.15.146:0/1666630464 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f1158157d80 con 0x7f1144003f30 2024-04-25T15:19:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f113c0744b0 msgr2=0x7f113c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f113c0744b0 0x7f113c0769c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1148033000 tx=0x7f1148008030 comp rx=0 tx=0).stop 2024-04-25T15:19:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 msgr2=0x7f1158158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1158158060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1154007770 tx=0x7f115400a360 comp rx=0 tx=0).stop 2024-04-25T15:19:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 shutdown_connections 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f113c0744b0 0x7f113c0769c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1158157430 0x7f1158157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1158155880 0x7f1158143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 --2- 172.21.15.146:0/1666630464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1144003f30 0x7f1158158060 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 >> 172.21.15.146:0/1666630464 conn(0x7f11580d5a50 msgr2=0x7f11580ba410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 shutdown_connections 2024-04-25T15:19:47.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.470+0000 7f1160100640 1 -- 172.21.15.146:0/1666630464 wait complete. 2024-04-25T15:19:47.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:19:47.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.602+0000 7f8bd0e53640 1 Processor -- start 2024-04-25T15:19:47.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.602+0000 7f8bd0e53640 1 -- start start 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.603+0000 7f8bd0e53640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 0x7f8bcc0d1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.603+0000 7f8bd0e53640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bcc0d2e90 0x7f8bcc0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.603+0000 7f8bd0e53640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bcc0b3d10 con 0x7f8bcc06f330 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.603+0000 7f8bd0e53640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8bcc06e490 con 0x7f8bcc0d2e90 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.603+0000 7f8bd0e53640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8bcc06e770 con 0x7f8bcc06def0 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 0x7f8bcc0d1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 0x7f8bcc0d1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:52004/0 (socket says 172.21.15.146:52004) 2024-04-25T15:19:47.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 -- 172.21.15.146:0/2279299927 learned_addr learned my addr 172.21.15.146:0/2279299927 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bcad9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bcc0d2e90 0x7f8bcc0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 -- 172.21.15.146:0/2279299927 >> v2:172.21.15.177:3300/0 conn(0x7f8bcc0d2e90 msgr2=0x7f8bcc0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 --2- 172.21.15.146:0/2279299927 >> v2:172.21.15.177:3300/0 conn(0x7f8bcc0d2e90 0x7f8bcc0b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 -- 172.21.15.146:0/2279299927 >> v1:172.21.15.177:6789/0 conn(0x7f8bcc06f330 legacy=0x7f8bcc0d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bca59c640 1 -- 172.21.15.146:0/2279299927 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bcc06ea50 con 0x7f8bcc06def0 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.604+0000 7f8bcad9d640 1 --2- 172.21.15.146:0/2279299927 >> v2:172.21.15.177:3300/0 conn(0x7f8bcc0d2e90 0x7f8bcc0b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:47.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.605+0000 7f8bca59c640 1 --2- 172.21.15.146:0/2279299927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 0x7f8bcc0d1ee0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009e50 tx=0x7f8bc402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ab0b88585b957d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.605+0000 7f8bc959a640 1 -- 172.21.15.146:0/2279299927 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4036070 con 0x7f8bcc06def0 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.605+0000 7f8bc959a640 1 -- 172.21.15.146:0/2279299927 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc4031600 con 0x7f8bcc06def0 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.605+0000 7f8bc959a640 1 -- 172.21.15.146:0/2279299927 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc40319d0 con 0x7f8bcc06def0 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 -- 172.21.15.146:0/2279299927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 msgr2=0x7f8bcc0d1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/2279299927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 0x7f8bcc0d1ee0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009e50 tx=0x7f8bc402a710 comp rx=0 tx=0).stop 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 -- 172.21.15.146:0/2279299927 shutdown_connections 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/2279299927 >> v2:172.21.15.177:3300/0 conn(0x7f8bcc0d2e90 0x7f8bcc0b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/2279299927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc06def0 0x7f8bcc0d1ee0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 -- 172.21.15.146:0/2279299927 >> 172.21.15.146:0/2279299927 conn(0x7f8bcc0b74e0 msgr2=0x7f8bcc0b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 -- 172.21.15.146:0/2279299927 shutdown_connections 2024-04-25T15:19:47.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.606+0000 7f8bd0e53640 1 -- 172.21.15.146:0/2279299927 wait complete. 2024-04-25T15:19:47.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.608+0000 7f8bd0e53640 1 Processor -- start 2024-04-25T15:19:47.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.608+0000 7f8bd0e53640 1 -- start start 2024-04-25T15:19:47.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bd0e53640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 0x7f8bcc150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:47.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bd0e53640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bcc06f330 0x7f8bcc0aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bd0e53640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc1e9a70 0x7f8bcc1e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bd0e53640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8bcc06f8c0 con 0x7f8bcc06def0 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bd0e53640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8bcc06f740 con 0x7f8bcc06f330 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bd0e53640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8bcc0d03c0 con 0x7f8bcc1e9a70 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 0x7f8bcc150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 0x7f8bcc150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36164/0 (socket says 172.21.15.146:36164) 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 -- 172.21.15.146:0/924950116 learned_addr learned my addr 172.21.15.146:0/924950116 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bcad9d640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc1e9a70 0x7f8bcc1e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bc9d9b640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bcc06f330 0x7f8bcc0aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 -- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc1e9a70 msgr2=0x7f8bcc1e9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc1e9a70 0x7f8bcc1e9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 -- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bcc06f330 msgr2=0x7f8bcc0aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bcc06f330 0x7f8bcc0aff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:47.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bca59c640 1 -- 172.21.15.146:0/924950116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bcc151090 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.609+0000 7f8bc9d9b640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bcc06f330 0x7f8bcc0aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.610+0000 7f8bca59c640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 0x7f8bcc150b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4000f80 tx=0x7f8bc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.610+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4036050 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.610+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc4042070 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.610+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bcc1ea3c0 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.610+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4009510 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.610+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bcc0b0cb0 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.611+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bcc0a08a0 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.611+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8bc4007c70 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.612+0000 7f8bbaffd640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.612+0000 7f8bc9d9b640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.612+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f8bc40b60e0 con 0x7f8bcc06def0 2024-04-25T15:19:47.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.612+0000 7f8bc9d9b640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc1513b0 tx=0x7f8bc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:47.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.616+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8bcc1ea3c0 con 0x7f8bcc06def0 2024-04-25T15:19:47.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:47.793+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8bcc050820 con 0x7f8bcc06def0 2024-04-25T15:19:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.478+0000 7f8bbaffd640 1 -- 172.21.15.146:0/924950116 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f8bcc151090 con 0x7f8bcc06def0 2024-04-25T15:19:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data_ec' created 2024-04-25T15:19:49.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.483+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 msgr2=0x7f8bac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:49.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.483+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc1513b0 tx=0x7f8bc0009040 comp rx=0 tx=0).stop 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.483+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 msgr2=0x7f8bcc150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.483+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 0x7f8bcc150b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4000f80 tx=0x7f8bc4004a40 comp rx=0 tx=0).stop 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.484+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 shutdown_connections 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.484+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.484+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bcc1e9a70 0x7f8bcc1e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.484+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bcc06f330 0x7f8bcc0aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.484+0000 7f8bd0e53640 1 --2- 172.21.15.146:0/924950116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bcc06def0 0x7f8bcc150b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.484+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 >> 172.21.15.146:0/924950116 conn(0x7f8bcc0b74e0 msgr2=0x7f8bcc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:49.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.485+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 shutdown_connections 2024-04-25T15:19:49.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.486+0000 7f8bd0e53640 1 -- 172.21.15.146:0/924950116 wait complete. 2024-04-25T15:19:49.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T15:19:49.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.613+0000 7fe91c34d640 1 Processor -- start 2024-04-25T15:19:49.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.613+0000 7fe91c34d640 1 -- start start 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91c34d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 0x7fe91406eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91c34d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe91406f060 0x7fe914160660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91c34d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91408cb70 con 0x7fe914160ba0 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91c34d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe914161380 con 0x7fe91406f060 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91c34d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe914161660 con 0x7fe91408d600 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91a8ea640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe914160ba0 0x7fe9140d1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:57916/0 (socket says 172.21.15.146:57916) 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91a8ea640 1 -- 172.21.15.146:0/467019187 learned_addr learned my addr 172.21.15.146:0/467019187 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9198e8640 1 --2- 172.21.15.146:0/467019187 >> v2:172.21.15.177:3300/0 conn(0x7fe91406f060 0x7fe914160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe91a0e9640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 0x7fe91406eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085655433 0 0) 0x7fe91408cb70 con 0x7fe914160ba0 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe900003520 con 0x7fe914160ba0 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1405204906 0 0) 0x7fe900003520 con 0x7fe914160ba0 2024-04-25T15:19:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 msgr2=0x7fe91406eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 0x7fe91406eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 >> v2:172.21.15.177:3300/0 conn(0x7fe91406f060 msgr2=0x7fe914160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 --2- 172.21.15.146:0/467019187 >> v2:172.21.15.177:3300/0 conn(0x7fe91406f060 0x7fe914160660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91408ce00 con 0x7fe914160ba0 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fe90c001670 con 0x7fe914160ba0 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 >> v1:172.21.15.146:6789/0 conn(0x7fe914160ba0 legacy=0x7fe9140d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 0x7fe9000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.615+0000 7fe9190e7640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900003f30 0x7fe900004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe9190e7640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe900004d40 0x7fe900005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe914161380 con 0x7fe900004d40 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe914161660 con 0x7fe900003f30 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe90000bac0 con 0x7fe91408d600 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe9198e8640 1 --2- 172.21.15.146:0/467019187 >> v2:172.21.15.177:3300/0 conn(0x7fe91406f060 0x7fe914160660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a8ea640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe900004d40 0x7fe900005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a0e9640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 0x7fe9000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe9198e8640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900003f30 0x7fe900004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a8ea640 1 -- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 msgr2=0x7fe9000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a8ea640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe91408d600 0x7fe9000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a8ea640 1 -- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900003f30 msgr2=0x7fe900004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a8ea640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900003f30 0x7fe900004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.616+0000 7fe91a8ea640 1 -- 172.21.15.146:0/467019187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe900003520 con 0x7fe900004d40 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe91a8ea640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe900004d40 0x7fe900005150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00c210 tx=0x7fe900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701f3ac8641d52b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c02e070 con 0x7fe900004d40 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90c005ad0 con 0x7fe900004d40 2024-04-25T15:19:49.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe9190e7640 1 -- 172.21.15.146:0/467019187 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c007d60 con 0x7fe900004d40 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe91c34d640 1 -- 172.21.15.146:0/467019187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe900004d40 msgr2=0x7fe900005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe91c34d640 1 --2- 172.21.15.146:0/467019187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe900004d40 0x7fe900005150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00c210 tx=0x7fe900005830 comp rx=0 tx=0).stop 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe91a8ea640 1 -- 172.21.15.146:0/467019187 reap_dead start 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe91c34d640 1 -- 172.21.15.146:0/467019187 shutdown_connections 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.617+0000 7fe91c34d640 1 -- 172.21.15.146:0/467019187 >> 172.21.15.146:0/467019187 conn(0x7fe9140b52f0 msgr2=0x7fe9140b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.618+0000 7fe91c34d640 1 -- 172.21.15.146:0/467019187 shutdown_connections 2024-04-25T15:19:49.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.618+0000 7fe91c34d640 1 -- 172.21.15.146:0/467019187 wait complete. 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.619+0000 7fe91c34d640 1 Processor -- start 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.619+0000 7fe91c34d640 1 -- start start 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900004d40 0x7fe9141506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 0x7fe9140b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9140b1e70 0x7fe9140b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe914161130 con 0x7fe9140b81b0 2024-04-25T15:19:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe914160fb0 con 0x7fe900004d40 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe9140b8d80 con 0x7fe9140b1e70 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 0x7fe9140b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 0x7fe9140b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36178/0 (socket says 172.21.15.146:36178) 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 -- 172.21.15.146:0/3620492007 learned_addr learned my addr 172.21.15.146:0/3620492007 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91a8ea640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9140b1e70 0x7fe9140b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91a0e9640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900004d40 0x7fe9141506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 -- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9140b1e70 msgr2=0x7fe9140b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9140b1e70 0x7fe9140b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 -- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900004d40 msgr2=0x7fe9141506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900004d40 0x7fe9141506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 -- 172.21.15.146:0/3620492007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9140b27c0 con 0x7fe9140b81b0 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe9198e8640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 0x7fe9140b85c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe904011cd0 tx=0x7fe90400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90400fd20 con 0x7fe9140b81b0 2024-04-25T15:19:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe904004020 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe914150190 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904004410 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.621+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9140b73f0 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.622+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe91406a0e0 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.623+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe904012040 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.623+0000 7fe90affd640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8f80744e0 0x7fe8f80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.624+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7fe904095bc0 con 0x7fe9140b81b0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.624+0000 7fe91a0e9640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8f80744e0 0x7fe8f80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.624+0000 7fe91a0e9640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8f80744e0 0x7fe8f80769f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe910007520 tx=0x7fe91000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.628+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9140b27c0 con 0x7fe9140b81b0 2024-04-25T15:19:49.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:49.805+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fe9140d2fe0 con 0x7fe9140b81b0 2024-04-25T15:19:50.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.483+0000 7fe90affd640 1 -- 172.21.15.146:0/3620492007 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7fe904019030 con 0x7fe9140b81b0 2024-04-25T15:19:50.484 INFO:teuthology.orchestra.run.smithi146.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8f80744e0 msgr2=0x7fe8f80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8f80744e0 0x7fe8f80769f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe910007520 tx=0x7fe91000b040 comp rx=0 tx=0).stop 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 msgr2=0x7fe9140b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 0x7fe9140b85c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe904011cd0 tx=0x7fe90400f610 comp rx=0 tx=0).stop 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 shutdown_connections 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8f80744e0 0x7fe8f80769f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9140b1e70 0x7fe9140b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9140b81b0 0x7fe9140b85c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 --2- 172.21.15.146:0/3620492007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900004d40 0x7fe9141506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 >> 172.21.15.146:0/3620492007 conn(0x7fe9140b52f0 msgr2=0x7fe9140b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 shutdown_connections 2024-04-25T15:19:50.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.486+0000 7fe91c34d640 1 -- 172.21.15.146:0/3620492007 wait complete. 2024-04-25T15:19:50.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T15:19:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.625+0000 7fe14c18f640 1 Processor -- start 2024-04-25T15:19:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.626+0000 7fe14c18f640 1 -- start start 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14c18f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe144072c80 0x7fe144168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14c18f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1440d2740 con 0x7fe1440cf0f0 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14c18f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1440cf500 con 0x7fe1440b2cd0 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14c18f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe14408da90 con 0x7fe144072c80 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14972a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe1440cf0f0 0x7fe1440d2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54084/0 (socket says 172.21.15.146:54084) 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14972a640 1 -- 172.21.15.146:0/1974600370 learned_addr learned my addr 172.21.15.146:0/1974600370 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 667115050 0 0) 0x7fe1440d2740 con 0x7fe1440cf0f0 2024-04-25T15:19:50.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe130003520 con 0x7fe1440cf0f0 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.627+0000 7fe14a72c640 1 --2- 172.21.15.146:0/1974600370 >> v2:172.21.15.177:3300/0 conn(0x7fe144072c80 0x7fe144168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2969121940 0 0) 0x7fe130003520 con 0x7fe1440cf0f0 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 >> v2:172.21.15.177:3300/0 conn(0x7fe144072c80 msgr2=0x7fe144168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 --2- 172.21.15.146:0/1974600370 >> v2:172.21.15.177:3300/0 conn(0x7fe144072c80 0x7fe144168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 >> v1:172.21.15.177:6789/0 conn(0x7fe1440b2cd0 legacy=0x7fe1440d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1440cf7d0 con 0x7fe1440cf0f0 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fe134001670 con 0x7fe1440cf0f0 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 >> v1:172.21.15.146:6789/0 conn(0x7fe1440cf0f0 legacy=0x7fe1440d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1300039b0 0x7fe130003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe130004800 0x7fe130004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130005250 0x7fe13000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe13000c1b0 con 0x7fe130004800 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe13000c360 con 0x7fe130005250 2024-04-25T15:19:50.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe13000c560 con 0x7fe1300039b0 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14972a640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe130004800 0x7fe130004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14a72c640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130005250 0x7fe13000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe149f2b640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1300039b0 0x7fe130003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14972a640 1 -- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1300039b0 msgr2=0x7fe130003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14972a640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1300039b0 0x7fe130003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14972a640 1 -- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130005250 msgr2=0x7fe13000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14972a640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130005250 0x7fe13000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14a72c640 1 -- 172.21.15.146:0/1974600370 reap_dead start 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.628+0000 7fe14972a640 1 -- 172.21.15.146:0/1974600370 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe130003520 con 0x7fe130004800 2024-04-25T15:19:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.629+0000 7fe14972a640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe130004800 0x7fe130004c10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe1440d48b0 tx=0x7fe130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6c830d0a6edbb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.629+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134007d30 con 0x7fe130004800 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.629+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe134005af0 con 0x7fe130004800 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.629+0000 7fe148f29640 1 -- 172.21.15.146:0/1974600370 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134007070 con 0x7fe130004800 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.629+0000 7fe14c18f640 1 -- 172.21.15.146:0/1974600370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe130004800 msgr2=0x7fe130004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.629+0000 7fe14c18f640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe130004800 0x7fe130004c10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe1440d48b0 tx=0x7fe130005830 comp rx=0 tx=0).stop 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.630+0000 7fe14c18f640 1 -- 172.21.15.146:0/1974600370 shutdown_connections 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.630+0000 7fe14c18f640 1 --2- 172.21.15.146:0/1974600370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe130004800 0x7fe130004c10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.630+0000 7fe14c18f640 1 -- 172.21.15.146:0/1974600370 >> 172.21.15.146:0/1974600370 conn(0x7fe1440b52e0 msgr2=0x7fe1440b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.630+0000 7fe14c18f640 1 -- 172.21.15.146:0/1974600370 shutdown_connections 2024-04-25T15:19:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.630+0000 7fe14c18f640 1 -- 172.21.15.146:0/1974600370 wait complete. 2024-04-25T15:19:50.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.631+0000 7fe14c18f640 1 Processor -- start 2024-04-25T15:19:50.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.631+0000 7fe14c18f640 1 -- start start 2024-04-25T15:19:50.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130004800 0x7fe144153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1440ca710 0x7fe1440cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 0x7fe1440ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440717e0 con 0x7fe1440ccb60 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440d0070 con 0x7fe130004800 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe1440cfd90 con 0x7fe1440ca710 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 0x7fe1440ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 0x7fe1440ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36520/0 (socket says 172.21.15.146:36520) 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 -- 172.21.15.146:0/3201993835 learned_addr learned my addr 172.21.15.146:0/3201993835 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14972a640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1440ca710 0x7fe1440cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe149f2b640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130004800 0x7fe144153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 -- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1440ca710 msgr2=0x7fe1440cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1440ca710 0x7fe1440cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 -- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130004800 msgr2=0x7fe144153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:50.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130004800 0x7fe144153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 -- 172.21.15.146:0/3201993835 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1440cd4b0 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe149f2b640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130004800 0x7fe144153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14a72c640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 0x7fe1440ccf70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe144071790 tx=0x7fe13c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c004030 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13c01b070 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1440cb480 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c004430 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.633+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1440cba40 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.635+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1440cb1f0 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.635+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe13c012040 con 0x7fe1440ccb60 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.635+0000 7fe13a7fc640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.635+0000 7fe149f2b640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.635+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fe13c0954b0 con 0x7fe1440ccb60 2024-04-25T15:19:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.636+0000 7fe149f2b640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe140007fd0 tx=0x7fe140009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.640+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe1440cd4b0 con 0x7fe1440ccb60 2024-04-25T15:19:50.738 INFO:tasks.ceph.mds.b.smithi177.stderr:2024-04-25T15:19:50.737+0000 7fa569503640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:19:50.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:50.852+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe14406a0e0 con 0x7fe1440ccb60 2024-04-25T15:19:51.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.485+0000 7fe13a7fc640 1 -- 172.21.15.146:0/3201993835 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7fe13c05f910 con 0x7fe1440ccb60 2024-04-25T15:19:51.486 INFO:teuthology.orchestra.run.smithi146.stderr:added data pool 4 to fsmap 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 msgr2=0x7fe1280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe140007fd0 tx=0x7fe140009040 comp rx=0 tx=0).stop 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 msgr2=0x7fe1440ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 0x7fe1440ccf70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe144071790 tx=0x7fe13c011cf0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 shutdown_connections 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe1280744b0 0x7fe1280769c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1440ccb60 0x7fe1440ccf70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1440ca710 0x7fe1440cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 --2- 172.21.15.146:0/3201993835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe130004800 0x7fe144153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 >> 172.21.15.146:0/3201993835 conn(0x7fe1440b52e0 msgr2=0x7fe14408c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:51.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 shutdown_connections 2024-04-25T15:19:51.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.488+0000 7fe14c18f640 1 -- 172.21.15.146:0/3201993835 wait complete. 2024-04-25T15:19:51.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:19:51.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.611+0000 7f0b2359e640 1 Processor -- start 2024-04-25T15:19:51.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.611+0000 7f0b2359e640 1 -- start start 2024-04-25T15:19:51.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.612+0000 7f0b2359e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0b240b8dd0 0x7f0b2415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.612+0000 7f0b2359e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b240b49f0 0x7f0b241639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b2359e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b240d4e20 con 0x7f0b24163ee0 2024-04-25T15:19:51.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b2359e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0b240d4bd0 con 0x7f0b240b49f0 2024-04-25T15:19:51.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b2359e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0b2406c550 con 0x7f0b240b8dd0 2024-04-25T15:19:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b240b49f0 0x7f0b241639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b240b49f0 0x7f0b241639a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36536/0 (socket says 172.21.15.146:36536) 2024-04-25T15:19:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 -- 172.21.15.146:0/3044450762 learned_addr learned my addr 172.21.15.146:0/3044450762 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b22d9d640 1 --1- 172.21.15.146:0/3044450762 >> v1:172.21.15.146:6789/0 conn(0x7f0b24163ee0 0x7f0b240d0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54098/0 (socket says 172.21.15.146:54098) 2024-04-25T15:19:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b2259c640 1 --2- 172.21.15.146:0/3044450762 >> v2:172.21.15.177:3300/0 conn(0x7f0b240b8dd0 0x7f0b2415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 -- 172.21.15.146:0/3044450762 >> v2:172.21.15.177:3300/0 conn(0x7f0b240b8dd0 msgr2=0x7f0b2415fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 --2- 172.21.15.146:0/3044450762 >> v2:172.21.15.177:3300/0 conn(0x7f0b240b8dd0 0x7f0b2415fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 -- 172.21.15.146:0/3044450762 >> v1:172.21.15.146:6789/0 conn(0x7f0b24163ee0 legacy=0x7f0b240d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b21d9b640 1 -- 172.21.15.146:0/3044450762 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b240d5100 con 0x7f0b240b49f0 2024-04-25T15:19:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.613+0000 7f0b2159a640 1 -- 172.21.15.146:0/3044450762 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110697281 0 0) 0x7f0b240d4e20 con 0x7f0b24163ee0 2024-04-25T15:19:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.614+0000 7f0b21d9b640 1 --2- 172.21.15.146:0/3044450762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b240b49f0 0x7f0b241639a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009870 tx=0x7f0b1802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4adc28e6506464 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.614+0000 7f0b2159a640 1 -- 172.21.15.146:0/3044450762 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18037070 con 0x7f0b240b49f0 2024-04-25T15:19:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.614+0000 7f0b2159a640 1 -- 172.21.15.146:0/3044450762 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b18032440 con 0x7f0b240b49f0 2024-04-25T15:19:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.614+0000 7f0b2159a640 1 -- 172.21.15.146:0/3044450762 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b18032810 con 0x7f0b240b49f0 2024-04-25T15:19:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.614+0000 7f0b2359e640 1 -- 172.21.15.146:0/3044450762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b240b49f0 msgr2=0x7f0b241639a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.614+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3044450762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b240b49f0 0x7f0b241639a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009870 tx=0x7f0b1802a2a0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.615+0000 7f0b2359e640 1 -- 172.21.15.146:0/3044450762 shutdown_connections 2024-04-25T15:19:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.615+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3044450762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b240b49f0 0x7f0b241639a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.615+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3044450762 >> v2:172.21.15.177:3300/0 conn(0x7f0b240b8dd0 0x7f0b2415fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.615+0000 7f0b2359e640 1 -- 172.21.15.146:0/3044450762 >> 172.21.15.146:0/3044450762 conn(0x7f0b240d62f0 msgr2=0x7f0b240d2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.615+0000 7f0b2359e640 1 -- 172.21.15.146:0/3044450762 shutdown_connections 2024-04-25T15:19:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.615+0000 7f0b2359e640 1 -- 172.21.15.146:0/3044450762 wait complete. 2024-04-25T15:19:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.616+0000 7f0b2359e640 1 Processor -- start 2024-04-25T15:19:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.617+0000 7f0b2359e640 1 -- start start 2024-04-25T15:19:51.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2359e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b240b49f0 0x7f0b24161940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2359e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b240b8dd0 0x7f0b241e9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 0x7f0b240acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2359e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0b241602f0 con 0x7f0b24163ee0 2024-04-25T15:19:51.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2359e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2406cb10 con 0x7f0b240b8dd0 2024-04-25T15:19:51.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2359e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0b241642f0 con 0x7f0b240b49f0 2024-04-25T15:19:51.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 0x7f0b240acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 0x7f0b240acc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36548/0 (socket says 172.21.15.146:36548) 2024-04-25T15:19:51.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 -- 172.21.15.146:0/3393971458 learned_addr learned my addr 172.21.15.146:0/3393971458 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:51.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b2259c640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b240b49f0 0x7f0b24161940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b21d9b640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b240b8dd0 0x7f0b241e9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 -- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b240b49f0 msgr2=0x7f0b24161940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b240b49f0 0x7f0b24161940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 -- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b240b8dd0 msgr2=0x7f0b241e9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b240b8dd0 0x7f0b241e9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 -- 172.21.15.146:0/3393971458 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b241ea930 con 0x7f0b24163ee0 2024-04-25T15:19:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b21d9b640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b240b8dd0 0x7f0b241e9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.618+0000 7f0b22d9d640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 0x7f0b240acc30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c780 tx=0x7f0b1c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.619+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c018070 con 0x7f0b24163ee0 2024-04-25T15:19:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.619+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b1c00ce30 con 0x7f0b24163ee0 2024-04-25T15:19:51.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.619+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b241eac10 con 0x7f0b24163ee0 2024-04-25T15:19:51.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.619+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00a840 con 0x7f0b24163ee0 2024-04-25T15:19:51.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.619+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b24159c90 con 0x7f0b24163ee0 2024-04-25T15:19:51.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.620+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b24050740 con 0x7f0b24163ee0 2024-04-25T15:19:51.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.620+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0b1c01b070 con 0x7f0b24163ee0 2024-04-25T15:19:51.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.621+0000 7f0b12ffd640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b040744b0 0x7f0b040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.621+0000 7f0b2259c640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b040744b0 0x7f0b040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.621+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0b1c094af0 con 0x7f0b24163ee0 2024-04-25T15:19:51.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.621+0000 7f0b2259c640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b040744b0 0x7f0b040769c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b240d3cb0 tx=0x7f0b14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:51.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.626+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b1c05ef40 con 0x7f0b24163ee0 2024-04-25T15:19:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.841+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b240d3ce0 con 0x7f0b24163ee0 2024-04-25T15:19:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.841+0000 7f0b12ffd640 1 -- 172.21.15.146:0/3393971458 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5189 (secure 0 0 0) 0x7f0b1c05e840 con 0x7f0b24163ee0 2024-04-25T15:19:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:51.842 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:19:50.853159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:19:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 6 2024-04-25T15:19:51.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b040744b0 msgr2=0x7f0b040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b040744b0 0x7f0b040769c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b240d3cb0 tx=0x7f0b14008040 comp rx=0 tx=0).stop 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 msgr2=0x7f0b240acc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 0x7f0b240acc30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c780 tx=0x7f0b1c00cc40 comp rx=0 tx=0).stop 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 shutdown_connections 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b040744b0 0x7f0b040769c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b24163ee0 0x7f0b240acc30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b240b8dd0 0x7f0b241e9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 --2- 172.21.15.146:0/3393971458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b240b49f0 0x7f0b24161940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.843+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 >> 172.21.15.146:0/3393971458 conn(0x7f0b240d62f0 msgr2=0x7f0b240b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.844+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 shutdown_connections 2024-04-25T15:19:51.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.844+0000 7f0b2359e640 1 -- 172.21.15.146:0/3393971458 wait complete. 2024-04-25T15:19:51.855 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:51.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.962+0000 7f7c5291f640 1 Processor -- start 2024-04-25T15:19:51.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.962+0000 7f7c5291f640 1 -- start start 2024-04-25T15:19:51.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c5291f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c1622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c5291f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c4c0ce020 0x7f7c4c0b6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c5291f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4c0ce8c0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c5291f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f7c4c0d1950 con 0x7f7c4c0ce020 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c5291f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c0ceba0 con 0x7f7c4c15def0 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c4c0ce020 0x7f7c4c0b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7c4c0ce020 0x7f7c4c0b6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36564/0 (socket says 172.21.15.146:36564) 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 -- 172.21.15.146:0/2527564415 learned_addr learned my addr 172.21.15.146:0/2527564415 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4bfff640 1 --2- 172.21.15.146:0/2527564415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c1622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 -- 172.21.15.146:0/2527564415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 msgr2=0x7f7c4c1622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 --2- 172.21.15.146:0/2527564415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c1622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 -- 172.21.15.146:0/2527564415 >> v1:172.21.15.177:6789/0 conn(0x7f7c4c0b5240 legacy=0x7f7c4c0b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.963+0000 7f7c4b7fe640 1 -- 172.21.15.146:0/2527564415 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c0cee80 con 0x7f7c4c0ce020 2024-04-25T15:19:51.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.964+0000 7f7c4b7fe640 1 --2- 172.21.15.146:0/2527564415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c0b6980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009870 tx=0x7f7c4002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36bc28201b184cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:51.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.964+0000 7f7c4affd640 1 -- 172.21.15.146:0/2527564415 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40037070 con 0x7f7c4c0ce020 2024-04-25T15:19:51.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.964+0000 7f7c4affd640 1 -- 172.21.15.146:0/2527564415 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c40032440 con 0x7f7c4c0ce020 2024-04-25T15:19:51.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.964+0000 7f7c4affd640 1 -- 172.21.15.146:0/2527564415 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40032810 con 0x7f7c4c0ce020 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 -- 172.21.15.146:0/2527564415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0ce020 msgr2=0x7f7c4c0b6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 --2- 172.21.15.146:0/2527564415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c0b6980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009870 tx=0x7f7c4002a270 comp rx=0 tx=0).stop 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 -- 172.21.15.146:0/2527564415 shutdown_connections 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 --2- 172.21.15.146:0/2527564415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c0b6980 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 --2- 172.21.15.146:0/2527564415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c1622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 -- 172.21.15.146:0/2527564415 >> 172.21.15.146:0/2527564415 conn(0x7f7c4c0d0240 msgr2=0x7f7c4c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 -- 172.21.15.146:0/2527564415 shutdown_connections 2024-04-25T15:19:51.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.965+0000 7f7c5291f640 1 -- 172.21.15.146:0/2527564415 wait complete. 2024-04-25T15:19:51.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.967+0000 7f7c5291f640 1 Processor -- start 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.967+0000 7f7c5291f640 1 -- start start 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c5291f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 0x7f7c4c0c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c5291f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c5291f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c5291f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c0b6ec0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c5291f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c0d23a0 con 0x7f7c4c0ce020 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c5291f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c0b7040 con 0x7f7c4c15def0 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 0x7f7c4c0c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 0x7f7c4c0c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36570/0 (socket says 172.21.15.146:36570) 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c4bfff640 1 -- 172.21.15.146:0/1403520225 learned_addr learned my addr 172.21.15.146:0/1403520225 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c4b7fe640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.968+0000 7f7c50ebc640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c4bfff640 1 -- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 msgr2=0x7f7c4c147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c4bfff640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c4bfff640 1 -- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c4c0ce020 msgr2=0x7f7c4c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c4bfff640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c4bfff640 1 -- 172.21.15.146:0/1403520225 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c0ca160 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c4bfff640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 0x7f7c4c0c9c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00d620 tx=0x7f7c3c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c00b620 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c019070 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c1466f0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c00ba00 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.969+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c146cb0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.970+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c06a0e0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.971+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7c3c00bba0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.971+0000 7f7c48ff9640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7c380744e0 0x7f7c380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.971+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7c3c0957c0 con 0x7f7c4c0b5240 2024-04-25T15:19:51.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.971+0000 7f7c4b7fe640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7c380744e0 0x7f7c380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:51.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.972+0000 7f7c4b7fe640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7c380744e0 0x7f7c380769f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c40034000 tx=0x7f7c4002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:51.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:51.976+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c4c0ca160 con 0x7f7c4c0b5240 2024-04-25T15:19:52.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.150+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c4c0b9700 con 0x7f7c4c0b5240 2024-04-25T15:19:52.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.151+0000 7f7c48ff9640 1 -- 172.21.15.146:0/1403520225 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18575 (secure 0 0 0) 0x7f7c3c05f650 con 0x7f7c4c0b5240 2024-04-25T15:19:52.153 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:52.153 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:50.852927+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.153+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7c380744e0 msgr2=0x7f7c380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.153+0000 7f7c5291f640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7c380744e0 0x7f7c380769f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c40034000 tx=0x7f7c4002e000 comp rx=0 tx=0).stop 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.153+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 msgr2=0x7f7c4c0c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.153+0000 7f7c5291f640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 0x7f7c4c0c9c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00d620 tx=0x7f7c3c00dae0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 shutdown_connections 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7c380744e0 0x7f7c380769f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c4c15def0 0x7f7c4c147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c4c0ce020 0x7f7c4c145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 --2- 172.21.15.146:0/1403520225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7c4c0b5240 0x7f7c4c0c9c20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 >> 172.21.15.146:0/1403520225 conn(0x7f7c4c0d0240 msgr2=0x7f7c4c0b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 shutdown_connections 2024-04-25T15:19:52.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.154+0000 7f7c5291f640 1 -- 172.21.15.146:0/1403520225 wait complete. 2024-04-25T15:19:52.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:52.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.273+0000 7f8990884640 1 Processor -- start 2024-04-25T15:19:52.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.273+0000 7f8990884640 1 -- start start 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f8990884640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0d0e10 0x7f898c15ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f8990884640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f898c0b45d0 0x7f898c06cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f8990884640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f898c0d5000 con 0x7f898c0d12a0 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f8990884640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f898c0d4db0 con 0x7f898c0b45d0 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f8990884640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f898c0d52e0 con 0x7f898c0d0e10 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898ad9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f898c0b45d0 0x7f898c06cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898a59c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f898c0d12a0 0x7f898c0d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54106/0 (socket says 172.21.15.146:54106) 2024-04-25T15:19:52.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898ad9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f898c0b45d0 0x7f898c06cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36586/0 (socket says 172.21.15.146:36586) 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898a59c640 1 -- 172.21.15.146:0/3341821799 learned_addr learned my addr 172.21.15.146:0/3341821799 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f8989d9b640 1 --2- 172.21.15.146:0/3341821799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0d0e10 0x7f898c15ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898ad9d640 1 -- 172.21.15.146:0/3341821799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0d0e10 msgr2=0x7f898c15ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898ad9d640 1 --2- 172.21.15.146:0/3341821799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0d0e10 0x7f898c15ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898ad9d640 1 -- 172.21.15.146:0/3341821799 >> v1:172.21.15.146:6789/0 conn(0x7f898c0d12a0 legacy=0x7f898c0d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898ad9d640 1 -- 172.21.15.146:0/3341821799 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c06c5f0 con 0x7f898c0b45d0 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.275+0000 7f898959a640 1 -- 172.21.15.146:0/3341821799 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714764301 0 0) 0x7f898c0d5000 con 0x7f898c0d12a0 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.276+0000 7f898ad9d640 1 --2- 172.21.15.146:0/3341821799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0b45d0 0x7f898c06cdf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8984009d90 tx=0x7f898402a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b03e519ed9145a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.276+0000 7f898959a640 1 -- 172.21.15.146:0/3341821799 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984007740 con 0x7f898c0b45d0 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.276+0000 7f898959a640 1 -- 172.21.15.146:0/3341821799 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89840078e0 con 0x7f898c0b45d0 2024-04-25T15:19:52.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.276+0000 7f898959a640 1 -- 172.21.15.146:0/3341821799 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984007cb0 con 0x7f898c0b45d0 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.276+0000 7f8990884640 1 -- 172.21.15.146:0/3341821799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0b45d0 msgr2=0x7f898c06cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.276+0000 7f8990884640 1 --2- 172.21.15.146:0/3341821799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0b45d0 0x7f898c06cdf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8984009d90 tx=0x7f898402a530 comp rx=0 tx=0).stop 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.277+0000 7f8990884640 1 -- 172.21.15.146:0/3341821799 shutdown_connections 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.277+0000 7f8990884640 1 --2- 172.21.15.146:0/3341821799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0b45d0 0x7f898c06cdf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.277+0000 7f8990884640 1 --2- 172.21.15.146:0/3341821799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0d0e10 0x7f898c15ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.277+0000 7f8990884640 1 -- 172.21.15.146:0/3341821799 >> 172.21.15.146:0/3341821799 conn(0x7f898c0d63b0 msgr2=0x7f898c0d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.277+0000 7f8990884640 1 -- 172.21.15.146:0/3341821799 shutdown_connections 2024-04-25T15:19:52.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.277+0000 7f8990884640 1 -- 172.21.15.146:0/3341821799 wait complete. 2024-04-25T15:19:52.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.279+0000 7f8990884640 1 Processor -- start 2024-04-25T15:19:52.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.279+0000 7f8990884640 1 -- start start 2024-04-25T15:19:52.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8990884640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0b45d0 0x7f898c1477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8990884640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 0x7f898c14a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8990884640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f898c1f5ea0 0x7f898c163f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8990884640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f898c0b9090 con 0x7f898c0d12a0 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8990884640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f898c0b8f10 con 0x7f898c1f5ea0 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8990884640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f898c0b4b20 con 0x7f898c0b45d0 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 0x7f898c14a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 0x7f898c14a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36590/0 (socket says 172.21.15.146:36590) 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 -- 172.21.15.146:0/2328981889 learned_addr learned my addr 172.21.15.146:0/2328981889 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f898ad9d640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f898c1f5ea0 0x7f898c163f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f898a59c640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0b45d0 0x7f898c1477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 -- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0b45d0 msgr2=0x7f898c1477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0b45d0 0x7f898c1477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 -- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f898c1f5ea0 msgr2=0x7f898c163f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f898c1f5ea0 0x7f898c163f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 -- 172.21.15.146:0/2328981889 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c162eb0 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f898a59c640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0b45d0 0x7f898c1477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.280+0000 7f8989d9b640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 0x7f898c14a1b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f898000e870 tx=0x7f898000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.281+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980017070 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.281+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f898000c940 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.281+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898c163190 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.281+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898000cd30 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.281+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898c163750 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.282+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f898c06a0e0 con 0x7f898c0d12a0 2024-04-25T15:19:52.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.282+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f898001a070 con 0x7f898c0d12a0 2024-04-25T15:19:52.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.283+0000 7f897affd640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f896c0744b0 0x7f896c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.283+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8980095820 con 0x7f898c0d12a0 2024-04-25T15:19:52.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.283+0000 7f898a59c640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f896c0744b0 0x7f896c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.283+0000 7f898a59c640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f896c0744b0 0x7f896c0769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f897c000f60 tx=0x7f897c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.288+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f898005fc70 con 0x7f898c0d12a0 2024-04-25T15:19:52.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.464+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f898c0d3e30 con 0x7f898c0d12a0 2024-04-25T15:19:52.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.465+0000 7f897affd640 1 -- 172.21.15.146:0/2328981889 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18575 (secure 0 0 0) 0x7f898005f570 con 0x7f898c0d12a0 2024-04-25T15:19:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:52.467 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:50.852927+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.467+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f896c0744b0 msgr2=0x7f896c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.467+0000 7f8990884640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f896c0744b0 0x7f896c0769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f897c000f60 tx=0x7f897c009040 comp rx=0 tx=0).stop 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 msgr2=0x7f898c14a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 0x7f898c14a1b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f898000e870 tx=0x7f898000ed30 comp rx=0 tx=0).stop 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 shutdown_connections 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f896c0744b0 0x7f896c0769c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f898c1f5ea0 0x7f898c163f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f898c0d12a0 0x7f898c14a1b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 --2- 172.21.15.146:0/2328981889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f898c0b45d0 0x7f898c1477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 >> 172.21.15.146:0/2328981889 conn(0x7f898c0d63b0 msgr2=0x7f898c0b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 shutdown_connections 2024-04-25T15:19:52.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.468+0000 7f8990884640 1 -- 172.21.15.146:0/2328981889 wait complete. 2024-04-25T15:19:52.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:19:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.604+0000 7f66c411f640 1 Processor -- start 2024-04-25T15:19:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.604+0000 7f66c411f640 1 -- start start 2024-04-25T15:19:52.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c411f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f66bc0d21f0 0x7f66bc15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c411f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 0x7f66bc0b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c411f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66bc09c2a0 con 0x7f66bc06f2c0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c411f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f66bc0d1d60 con 0x7f66bc0d21f0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c411f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f66bc09bce0 con 0x7f66bc06f6d0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c16ba640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f66bc06f2c0 0x7f66bc0b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54118/0 (socket says 172.21.15.146:54118) 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c16ba640 1 -- 172.21.15.146:0/2793340623 learned_addr learned my addr 172.21.15.146:0/2793340623 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195356173 0 0) 0x7f66bc09c2a0 con 0x7f66bc06f2c0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c1ebb640 1 --2- 172.21.15.146:0/2793340623 >> v2:172.21.15.177:3300/0 conn(0x7f66bc0d21f0 0x7f66bc15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a8003520 con 0x7f66bc06f2c0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.605+0000 7f66c26bc640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 0x7f66bc0b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1358080036 0 0) 0x7f66a8003520 con 0x7f66bc06f2c0 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 msgr2=0x7f66bc0b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 0x7f66bc0b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 >> v2:172.21.15.177:3300/0 conn(0x7f66bc0d21f0 msgr2=0x7f66bc15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 --2- 172.21.15.146:0/2793340623 >> v2:172.21.15.177:3300/0 conn(0x7f66bc0d21f0 0x7f66bc15ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc09bfc0 con 0x7f66bc06f2c0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f66ac001670 con 0x7f66bc06f2c0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 >> v1:172.21.15.146:6789/0 conn(0x7f66bc06f2c0 legacy=0x7f66bc0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66a80039b0 0x7f66a8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66a8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 0x7f66a8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc0d1d60 con 0x7f66a8004800 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc09bce0 con 0x7f66a80039b0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f66a800ba70 con 0x7f66bc06f6d0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c16ba640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66a8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c1ebb640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66a80039b0 0x7f66a8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c26bc640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 0x7f66a8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c16ba640 1 -- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 msgr2=0x7f66a8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c16ba640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc06f6d0 0x7f66a8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c16ba640 1 -- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66a80039b0 msgr2=0x7f66a8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c16ba640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66a80039b0 0x7f66a8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.606+0000 7f66c16ba640 1 -- 172.21.15.146:0/2793340623 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a8003520 con 0x7f66a8004800 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.607+0000 7f66c16ba640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66a8004c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f66bc072ef0 tx=0x7f66a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a765e4c187a7f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.607+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac007d50 con 0x7f66a8004800 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.607+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66ac0058d0 con 0x7f66a8004800 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.607+0000 7f66c0eb9640 1 -- 172.21.15.146:0/2793340623 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac005cc0 con 0x7f66a8004800 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c411f640 1 -- 172.21.15.146:0/2793340623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 msgr2=0x7f66a8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c411f640 1 --2- 172.21.15.146:0/2793340623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66a8004c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f66bc072ef0 tx=0x7f66a8005830 comp rx=0 tx=0).stop 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c26bc640 1 -- 172.21.15.146:0/2793340623 reap_dead start 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c411f640 1 -- 172.21.15.146:0/2793340623 shutdown_connections 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c411f640 1 -- 172.21.15.146:0/2793340623 >> 172.21.15.146:0/2793340623 conn(0x7f66bc0b7530 msgr2=0x7f66bc0b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c411f640 1 -- 172.21.15.146:0/2793340623 shutdown_connections 2024-04-25T15:19:52.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.608+0000 7f66c411f640 1 -- 172.21.15.146:0/2793340623 wait complete. 2024-04-25T15:19:52.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.610+0000 7f66c411f640 1 Processor -- start 2024-04-25T15:19:52.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.610+0000 7f66c411f640 1 -- start start 2024-04-25T15:19:52.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66bc148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66bc0ccff0 0x7f66bc0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc14b520 0x7f66bc14b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc09b580 con 0x7f66a8004800 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc09b400 con 0x7f66bc0ccff0 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f66bc0b3de0 con 0x7f66bc14b520 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66bc148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66bc148d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36622/0 (socket says 172.21.15.146:36622) 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 -- 172.21.15.146:0/4002727705 learned_addr learned my addr 172.21.15.146:0/4002727705 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 -- 172.21.15.146:0/4002727705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc14b520 msgr2=0x7f66bc14b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc14b520 0x7f66bc14b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 -- 172.21.15.146:0/4002727705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66bc0ccff0 msgr2=0x7f66bc0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66bc0ccff0 0x7f66bc0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 -- 172.21.15.146:0/4002727705 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc15d1b0 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c1ebb640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66bc148d00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009fc0 tx=0x7f66b800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8015430 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b80155d0 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66bc0ca220 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80159a0 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.611+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66bc0ca7e0 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.613+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66bc06a0e0 con 0x7f66a8004800 2024-04-25T15:19:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.613+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f66b8015b40 con 0x7f66a8004800 2024-04-25T15:19:52.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.613+0000 7f66b27fc640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f66a00744b0 0x7f66a00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.613+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f66b8096720 con 0x7f66a8004800 2024-04-25T15:19:52.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.613+0000 7f66c16ba640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f66a00744b0 0x7f66a00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.614+0000 7f66c16ba640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f66a00744b0 0x7f66a00769c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f66ac033000 tx=0x7f66ac007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.618+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66b8019030 con 0x7f66a8004800 2024-04-25T15:19:52.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.831+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66bc070860 con 0x7f66a8004800 2024-04-25T15:19:52.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.831+0000 7f66b27fc640 1 -- 172.21.15.146:0/4002727705 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5189 (secure 0 0 0) 0x7f66b8010040 con 0x7f66a8004800 2024-04-25T15:19:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:52.833 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:19:50.853159+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:19:52.833 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 6 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f66a00744b0 msgr2=0x7f66a00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f66a00744b0 0x7f66a00769c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f66ac033000 tx=0x7f66ac007000 comp rx=0 tx=0).stop 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 msgr2=0x7f66bc148d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66bc148d00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009fc0 tx=0x7f66b800f580 comp rx=0 tx=0).stop 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 shutdown_connections 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f66a00744b0 0x7f66a00769c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f66bc14b520 0x7f66bc14b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66bc0ccff0 0x7f66bc0cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 --2- 172.21.15.146:0/4002727705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66a8004800 0x7f66bc148d00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 >> 172.21.15.146:0/4002727705 conn(0x7f66bc0b7530 msgr2=0x7f66bc0b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:52.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 shutdown_connections 2024-04-25T15:19:52.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.834+0000 7f66c411f640 1 -- 172.21.15.146:0/4002727705 wait complete. 2024-04-25T15:19:52.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:52.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.952+0000 7fa367b57640 1 Processor -- start 2024-04-25T15:19:52.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.952+0000 7fa367b57640 1 -- start start 2024-04-25T15:19:52.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa367b57640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa360162890 0x7fa36008d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa367b57640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3600d1790 con 0x7fa3601622e0 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa367b57640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3600d1a70 con 0x7fa3600b4df0 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa367b57640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa3600d23a0 con 0x7fa360162890 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3658f3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa360162890 0x7fa36008d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3658f3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa360162890 0x7fa36008d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36626/0 (socket says 172.21.15.146:36626) 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3658f3640 1 -- 172.21.15.146:0/1617790315 learned_addr learned my addr 172.21.15.146:0/1617790315 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3658f3640 1 -- 172.21.15.146:0/1617790315 >> v1:172.21.15.177:6789/0 conn(0x7fa3600b4df0 legacy=0x7fa360161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3658f3640 1 -- 172.21.15.146:0/1617790315 >> v1:172.21.15.146:6789/0 conn(0x7fa3601622e0 legacy=0x7fa3600ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3658f3640 1 -- 172.21.15.146:0/1617790315 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3600d1d50 con 0x7fa360162890 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.954+0000 7fa3648f1640 1 -- 172.21.15.146:0/1617790315 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710180747 0 0) 0x7fa3600d1790 con 0x7fa3601622e0 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.955+0000 7fa3658f3640 1 --2- 172.21.15.146:0/1617790315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa36008d670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009850 tx=0x7fa35c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811b2b9339292177 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.955+0000 7fa3648f1640 1 -- 172.21.15.146:0/1617790315 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c004030 con 0x7fa360162890 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.955+0000 7fa3648f1640 1 -- 172.21.15.146:0/1617790315 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa35c0041d0 con 0x7fa360162890 2024-04-25T15:19:52.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.955+0000 7fa3648f1640 1 -- 172.21.15.146:0/1617790315 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c003610 con 0x7fa360162890 2024-04-25T15:19:52.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 -- 172.21.15.146:0/1617790315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 msgr2=0x7fa36008d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 --2- 172.21.15.146:0/1617790315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa36008d670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009850 tx=0x7fa35c02a250 comp rx=0 tx=0).stop 2024-04-25T15:19:52.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 -- 172.21.15.146:0/1617790315 shutdown_connections 2024-04-25T15:19:52.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 --2- 172.21.15.146:0/1617790315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa36008d670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 -- 172.21.15.146:0/1617790315 >> 172.21.15.146:0/1617790315 conn(0x7fa3600b52e0 msgr2=0x7fa3600b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:52.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 -- 172.21.15.146:0/1617790315 shutdown_connections 2024-04-25T15:19:52.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.956+0000 7fa367b57640 1 -- 172.21.15.146:0/1617790315 wait complete. 2024-04-25T15:19:52.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.958+0000 7fa367b57640 1 Processor -- start 2024-04-25T15:19:52.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.958+0000 7fa367b57640 1 -- start start 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa367b57640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3600b4df0 0x7fa360150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa367b57640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa3600b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa367b57640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3600b2140 0x7fa36009ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa367b57640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa36008dbb0 con 0x7fa360162890 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa367b57640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3600d2830 con 0x7fa3600b4df0 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa367b57640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa3600ceb10 con 0x7fa3600b2140 2024-04-25T15:19:52.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa3600b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa3600b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36630/0 (socket says 172.21.15.146:36630) 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 -- 172.21.15.146:0/4027604443 learned_addr learned my addr 172.21.15.146:0/4027604443 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3658f3640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3600b4df0 0x7fa360150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3660f4640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3600b2140 0x7fa36009ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 -- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3600b2140 msgr2=0x7fa36009ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3600b2140 0x7fa36009ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 -- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3600b4df0 msgr2=0x7fa360150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3600b4df0 0x7fa360150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 -- 172.21.15.146:0/4027604443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3600b3230 con 0x7fa360162890 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3650f2640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa3600b04c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c400 tx=0x7fa35000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.959+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350017070 con 0x7fa360162890 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.960+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3500052d0 con 0x7fa360162890 2024-04-25T15:19:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.960+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3600b10a0 con 0x7fa360162890 2024-04-25T15:19:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.960+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350003830 con 0x7fa360162890 2024-04-25T15:19:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.960+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3600b1660 con 0x7fa360162890 2024-04-25T15:19:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.960+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36006a0e0 con 0x7fa360162890 2024-04-25T15:19:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.961+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa35001a070 con 0x7fa360162890 2024-04-25T15:19:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.961+0000 7fa3567fc640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3440744b0 0x7fa3440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.962+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa350094880 con 0x7fa360162890 2024-04-25T15:19:52.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.962+0000 7fa3658f3640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3440744b0 0x7fa3440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:52.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.962+0000 7fa3658f3640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3440744b0 0x7fa3440769c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa35c033000 tx=0x7fa35c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:52.966+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3600b3230 con 0x7fa360162890 2024-04-25T15:19:53.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.142+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3600b61f0 con 0x7fa360162890 2024-04-25T15:19:53.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.144+0000 7fa3567fc640 1 -- 172.21.15.146:0/4027604443 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7fa35005e680 con 0x7fa360162890 2024-04-25T15:19:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:53.145 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3440744b0 msgr2=0x7fa3440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3440744b0 0x7fa3440769c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa35c033000 tx=0x7fa35c02e000 comp rx=0 tx=0).stop 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 msgr2=0x7fa3600b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa3600b04c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c400 tx=0x7fa35000c8c0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 shutdown_connections 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3440744b0 0x7fa3440769c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3600b2140 0x7fa36009ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa360162890 0x7fa3600b04c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 --2- 172.21.15.146:0/4027604443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3600b4df0 0x7fa360150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 >> 172.21.15.146:0/4027604443 conn(0x7fa3600b52e0 msgr2=0x7fa3600b8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 shutdown_connections 2024-04-25T15:19:53.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.146+0000 7fa367b57640 1 -- 172.21.15.146:0/4027604443 wait complete. 2024-04-25T15:19:53.561 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:19:45.824153+0000', 'modified': '2024-04-25T15:19:50.853159+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6834/3903573056', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 3903573056}, {'type': 'v1', 'addr': '172.21.15.177:6834', 'nonce': 3903573056}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-04-25T15:19:53.561 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:19:53.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T15:19:53.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.670+0000 7f5e39cc6640 1 Processor -- start 2024-04-25T15:19:53.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.670+0000 7f5e39cc6640 1 -- start start 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e39cc6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e39cc6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e340d0e10 0x7f5e3406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e39cc6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e340d5000 con 0x7f5e340b45d0 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e39cc6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5e340d4db0 con 0x7f5e340d0e10 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e39cc6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5e340d52e0 con 0x7f5e340d12a0 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e33fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e340d0e10 0x7f5e3406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e2affd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5e340b45d0 0x7f5e340d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54138/0 (socket says 172.21.15.146:54138) 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e33fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e340d0e10 0x7f5e3406cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36644/0 (socket says 172.21.15.146:36644) 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e2affd640 1 -- 172.21.15.146:0/822585386 learned_addr learned my addr 172.21.15.146:0/822585386 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e337fe640 1 --2- 172.21.15.146:0/822585386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e33fff640 1 -- 172.21.15.146:0/822585386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 msgr2=0x7f5e3415ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e33fff640 1 --2- 172.21.15.146:0/822585386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e33fff640 1 -- 172.21.15.146:0/822585386 >> v1:172.21.15.146:6789/0 conn(0x7f5e340b45d0 legacy=0x7f5e340d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e33fff640 1 -- 172.21.15.146:0/822585386 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3406c5f0 con 0x7f5e340d0e10 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.672+0000 7f5e32ffd640 1 -- 172.21.15.146:0/822585386 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166711462 0 0) 0x7f5e340d5000 con 0x7f5e340b45d0 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.673+0000 7f5e33fff640 1 --2- 172.21.15.146:0/822585386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340d0e10 0x7f5e3406cdf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009d90 tx=0x7f5e2c02a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d237f10b6487f7f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.673+0000 7f5e32ffd640 1 -- 172.21.15.146:0/822585386 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c007740 con 0x7f5e340d0e10 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.673+0000 7f5e32ffd640 1 -- 172.21.15.146:0/822585386 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e2c0078e0 con 0x7f5e340d0e10 2024-04-25T15:19:53.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.673+0000 7f5e32ffd640 1 -- 172.21.15.146:0/822585386 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c007cb0 con 0x7f5e340d0e10 2024-04-25T15:19:53.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 -- 172.21.15.146:0/822585386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340d0e10 msgr2=0x7f5e3406cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/822585386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340d0e10 0x7f5e3406cdf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009d90 tx=0x7f5e2c02a530 comp rx=0 tx=0).stop 2024-04-25T15:19:53.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 -- 172.21.15.146:0/822585386 shutdown_connections 2024-04-25T15:19:53.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/822585386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340d0e10 0x7f5e3406cdf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/822585386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 -- 172.21.15.146:0/822585386 >> 172.21.15.146:0/822585386 conn(0x7f5e340d63b0 msgr2=0x7f5e340d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 -- 172.21.15.146:0/822585386 shutdown_connections 2024-04-25T15:19:53.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.674+0000 7f5e39cc6640 1 -- 172.21.15.146:0/822585386 wait complete. 2024-04-25T15:19:53.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.676+0000 7f5e39cc6640 1 Processor -- start 2024-04-25T15:19:53.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.676+0000 7f5e39cc6640 1 -- start start 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e39cc6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 0x7f5e34153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e39cc6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e39cc6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e3415abc0 0x7f5e340cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e39cc6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5e340b9090 con 0x7f5e340b45d0 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e39cc6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e340b8f10 con 0x7f5e3415abc0 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e39cc6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5e340b4b20 con 0x7f5e340d12a0 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 0x7f5e34153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 0x7f5e34153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36650/0 (socket says 172.21.15.146:36650) 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 -- 172.21.15.146:0/2384233263 learned_addr learned my addr 172.21.15.146:0/2384233263 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e2affd640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e33fff640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e3415abc0 0x7f5e340cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 -- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 msgr2=0x7f5e3415a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 -- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e3415abc0 msgr2=0x7f5e340cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e3415abc0 0x7f5e340cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.677+0000 7f5e337fe640 1 -- 172.21.15.146:0/2384233263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e34154920 con 0x7f5e340b45d0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.678+0000 7f5e337fe640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 0x7f5e34153db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400d5d0 tx=0x7f5e2400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.678+0000 7f5e30ff9640 1 -- 172.21.15.146:0/2384233263 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24004240 con 0x7f5e340b45d0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.678+0000 7f5e30ff9640 1 -- 172.21.15.146:0/2384233263 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e240043e0 con 0x7f5e340b45d0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.678+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e34154c00 con 0x7f5e340b45d0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.678+0000 7f5e30ff9640 1 -- 172.21.15.146:0/2384233263 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24003830 con 0x7f5e340b45d0 2024-04-25T15:19:53.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.678+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e341551c0 con 0x7f5e340b45d0 2024-04-25T15:19:53.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.680+0000 7f5e30ff9640 1 -- 172.21.15.146:0/2384233263 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5e240039d0 con 0x7f5e340b45d0 2024-04-25T15:19:53.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.680+0000 7f5e30ff9640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e140744b0 0x7f5e140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.680+0000 7f5e2affd640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e140744b0 0x7f5e140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.680+0000 7f5e30ff9640 1 -- 172.21.15.146:0/2384233263 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5e240956c0 con 0x7f5e340b45d0 2024-04-25T15:19:53.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.680+0000 7f5e2affd640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e140744b0 0x7f5e140769c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e341f9af0 tx=0x7f5e1c007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.681+0000 7f5e40b76740 1 Processor -- start 2024-04-25T15:19:53.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.681+0000 7f5e40b76740 1 -- start start 2024-04-25T15:19:53.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.682+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0be2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.682+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0bdefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.682+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bb8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.682+0000 7f5e40b76740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55cfc0b1c740 con 0x55cfc07ed3b0 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.682+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfc0ab83a0 con 0x55cfc07f5590 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.682+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55cfc0b18700 con 0x55cfc0816270 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0bdefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0bdefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36666/0 (socket says 172.21.15.146:36666) 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 -- 172.21.15.146:0/3071578855 learned_addr learned my addr 172.21.15.146:0/3071578855 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e337fe640 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0be2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e33fff640 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bb8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 -- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 msgr2=0x55cfc0be2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0be2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 -- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 msgr2=0x55cfc0bb8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bb8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 -- 172.21.15.146:0/3071578855 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfc0ad4750 con 0x55cfc07ed3b0 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e33fff640 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bb8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e2affd640 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0bdefa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c004f60 tx=0x7f5e1c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f247b9b3bcc694b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e297fa640 1 -- 172.21.15.146:0/3071578855 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c026070 con 0x55cfc07ed3b0 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e297fa640 1 -- 172.21.15.146:0/3071578855 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1c001cf0 con 0x55cfc07ed3b0 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.683+0000 7f5e297fa640 1 -- 172.21.15.146:0/3071578855 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c013d80 con 0x55cfc07ed3b0 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 -- 172.21.15.146:0/3071578855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 msgr2=0x55cfc0bdefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0bdefa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c004f60 tx=0x7f5e1c013000 comp rx=0 tx=0).stop 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 -- 172.21.15.146:0/3071578855 shutdown_connections 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bb8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0bdefa0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 --2- 172.21.15.146:0/3071578855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0be2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 -- 172.21.15.146:0/3071578855 >> 172.21.15.146:0/3071578855 conn(0x55cfc080a3c0 msgr2=0x55cfc0bb79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 -- 172.21.15.146:0/3071578855 shutdown_connections 2024-04-25T15:19:53.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.684+0000 7f5e40b76740 1 -- 172.21.15.146:0/3071578855 wait complete. 2024-04-25T15:19:53.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.685+0000 7f5e40b76740 1 -- start start 2024-04-25T15:19:53.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.685+0000 7f5e40b76740 1 Processor -- start 2024-04-25T15:19:53.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc07ed3b0 0x55cfc0bdd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bdde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 0x55cfc0bda2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55cfc0aa3fe0 con 0x55cfc0816270 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfc0bdea60 con 0x55cfc07f5590 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55cfc0bfad40 con 0x55cfc07ed3b0 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e40b76740 10 client.0 _ll_register_callbacks cb 0x55cfc0aba150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e33fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 0x55cfc0bda2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e33fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 0x55cfc0bda2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36670/0 (socket says 172.21.15.146:36670) 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e33fff640 1 -- 172.21.15.146:0/1865123059 learned_addr learned my addr 172.21.15.146:0/1865123059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.686+0000 7f5e337fe640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc07ed3b0 0x55cfc0bdd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e2affd640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bdde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e33fff640 1 -- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc07ed3b0 msgr2=0x55cfc0bdd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e33fff640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc07ed3b0 0x55cfc0bdd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e33fff640 1 -- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 msgr2=0x55cfc0bdde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e33fff640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bdde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e33fff640 1 -- 172.21.15.146:0/1865123059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfc082e2d0 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e33fff640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 0x55cfc0bda2a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009eb0 tx=0x7f5e2c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc38acbe68ca38d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e127fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e127fc640 1 -- 172.21.15.146:0/1865123059 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c004390 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e127fc640 1 -- 172.21.15.146:0/1865123059 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5e2c004530 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e1c001cf0 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e127fc640 1 -- 172.21.15.146:0/1865123059 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f5e2c004730 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e40b76740 10 client.4428 fetch_fsmap learned FSMap version 6 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55cfc08f86b0 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e127fc640 1 -- 172.21.15.146:0/1865123059 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1c001cf0 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.687+0000 7f5e127fc640 1 -- 172.21.15.146:0/1865123059 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f5e2c007540 con 0x55cfc0816270 2024-04-25T15:19:53.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e40b76740 10 client.4428 fetch_fsmap finished waiting for FSMap version 6 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e40b76740 10 client.4428 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e40b76740 20 client.4428 populate_metadata read hostname 'smithi146' 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e40b76740 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc07d4e40 0x55cfc0c19fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e40b76740 4 client.4428 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 --> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfc0b2a350 con 0x55cfc07d4e40 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e337fe640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc07d4e40 0x55cfc0c19fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e127fc640 10 client.4428 ms_handle_connect on v2:172.21.15.177:6832/3903573056 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.688+0000 7f5e337fe640 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc07d4e40 0x55cfc0c19fc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=998e7ac74c05fd39 server_cookie=b38526ce075b66e1 in_seq=0 out_seq=0 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e127fc640 1 -- 172.21.15.146:0/1865123059 <== mds.0 v2:172.21.15.177:6832/3903573056 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cfc0b2a350 con 0x55cfc07d4e40 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e127fc640 10 client.4428 handle_command_reply: tid=0 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 client.4428 shutdown 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 msgr2=0x55cfc0bda2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 0x55cfc0bda2a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009eb0 tx=0x7f5e2c004ab0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 shutdown_connections 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc07d4e40 0x55cfc0c19fc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc0816270 0x55cfc0bda2a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bdde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 --2- 172.21.15.146:0/1865123059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc07ed3b0 0x55cfc0bdd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e127fc640 0 client.4428 ms_handle_reset on v2:172.21.15.177:6832/3903573056 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 >> 172.21.15.146:0/1865123059 conn(0x55cfc080a3c0 msgr2=0x55cfc0bb79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 shutdown_connections 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.689+0000 7f5e40b76740 1 -- 172.21.15.146:0/1865123059 wait complete. 2024-04-25T15:19:53.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.690+0000 7f5e40b76740 20 client.4428 trim_cache size 0 max 16384 2024-04-25T15:19:53.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.708+0000 7f5e40b76740 1 Processor -- start 2024-04-25T15:19:53.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.708+0000 7f5e40b76740 1 -- start start 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.709+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0c71f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.709+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bfb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.709+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bb1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.709+0000 7f5e40b76740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34202ea0 con 0x55cfc07ed3b0 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.709+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34202200 con 0x55cfc07f5590 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.709+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5e34202850 con 0x55cfc0816270 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0c71f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0c71f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36684/0 (socket says 172.21.15.146:36684) 2024-04-25T15:19:53.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 -- 172.21.15.146:0/849220340 learned_addr learned my addr 172.21.15.146:0/849220340 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e33fff640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bb1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e2affd640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bfb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 -- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 msgr2=0x55cfc0bb1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bb1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 -- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 msgr2=0x55cfc0bfb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bfb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 -- 172.21.15.146:0/849220340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfc0bb76e0 con 0x55cfc07ed3b0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e33fff640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bb1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e2affd640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bfb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e337fe640 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0c71f70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e2405c7c0 tx=0x7f5e24093780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4242309686005991 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e28ff9640 1 -- 172.21.15.146:0/849220340 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24063850 con 0x55cfc07ed3b0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e28ff9640 1 -- 172.21.15.146:0/849220340 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e24068270 con 0x55cfc07ed3b0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.710+0000 7f5e28ff9640 1 -- 172.21.15.146:0/849220340 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24003cc0 con 0x55cfc07ed3b0 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 -- 172.21.15.146:0/849220340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 msgr2=0x55cfc0c71f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0c71f70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e2405c7c0 tx=0x7f5e24093780 comp rx=0 tx=0).stop 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 -- 172.21.15.146:0/849220340 shutdown_connections 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bb1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0bfb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 --2- 172.21.15.146:0/849220340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0c71f70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 -- 172.21.15.146:0/849220340 >> 172.21.15.146:0/849220340 conn(0x55cfc080a3c0 msgr2=0x55cfc0be60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 -- 172.21.15.146:0/849220340 shutdown_connections 2024-04-25T15:19:53.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.711+0000 7f5e40b76740 1 -- 172.21.15.146:0/849220340 wait complete. 2024-04-25T15:19:53.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.712+0000 7f5e40b76740 1 -- start start 2024-04-25T15:19:53.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.712+0000 7f5e40b76740 1 Processor -- start 2024-04-25T15:19:53.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0be1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0c6de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bf7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55cfc0bde560 con 0x55cfc07ed3b0 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfc0bde7a0 con 0x55cfc07f5590 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55cfc0bdb8a0 con 0x55cfc0816270 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e40b76740 10 client.0 _ll_register_callbacks cb 0x7f5e2c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e337fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0be1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e337fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0be1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36694/0 (socket says 172.21.15.146:36694) 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e337fe640 1 -- 172.21.15.146:0/642998785 learned_addr learned my addr 172.21.15.146:0/642998785 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e33fff640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bf7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e2affd640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0c6de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.713+0000 7f5e337fe640 1 -- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 msgr2=0x55cfc0bf7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e337fe640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bf7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e337fe640 1 -- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 msgr2=0x55cfc0c6de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e337fe640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0c6de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e337fe640 1 -- 172.21.15.146:0/642998785 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfc0bb44b0 con 0x55cfc07ed3b0 2024-04-25T15:19:53.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e337fe640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0be1740 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e2405c7c0 tx=0x7f5e24095040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12cdb19cb18b882 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e127fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e127fc640 1 -- 172.21.15.146:0/642998785 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e240933a0 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e127fc640 1 -- 172.21.15.146:0/642998785 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5e24093540 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e24068270 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e127fc640 1 -- 172.21.15.146:0/642998785 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f5e24066b40 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 10 client.4434 fetch_fsmap learned FSMap version 6 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55cfc0bf5b80 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e127fc640 1 -- 172.21.15.146:0/642998785 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e24068270 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e127fc640 1 -- 172.21.15.146:0/642998785 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f5e24013070 con 0x55cfc07ed3b0 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 10 client.4434 fetch_fsmap finished waiting for FSMap version 6 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 10 client.4434 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 20 client.4434 populate_metadata read hostname 'smithi146' 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc0bb1300 0x55cfc0bb1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 4 client.4434 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:19:53.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 --> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] -- command(tid 0: {"prefix": "status"}) -- 0x55cfc0bffe40 con 0x55cfc0bb1300 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.714+0000 7f5e2affd640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc0bb1300 0x55cfc0bb1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e2affd640 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc0bb1300 0x55cfc0bb1710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b596ba011ec30ebe server_cookie=69b677e5ee1b0f2a in_seq=0 out_seq=0 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e127fc640 10 client.4434 ms_handle_connect on v2:172.21.15.177:6832/3903573056 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e127fc640 1 -- 172.21.15.146:0/642998785 <== mds.0 v2:172.21.15.177:6832/3903573056 1 ==== command_reply(tid 0: 0 ) ==== 8+0+323 (crc 0 0 0) 0x55cfc0bffe40 con 0x55cfc0bb1300 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e127fc640 10 client.4434 handle_command_reply: tid=0 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 client.4434 shutdown 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 msgr2=0x55cfc0be1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0be1740 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e2405c7c0 tx=0x7f5e24095040 comp rx=0 tx=0).stop 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 shutdown_connections 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x55cfc0bb1300 0x55cfc0bb1710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cfc0816270 0x55cfc0bf7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfc07f5590 0x55cfc0c6de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e127fc640 0 client.4434 ms_handle_reset on v2:172.21.15.177:6832/3903573056 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 --2- 172.21.15.146:0/642998785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cfc07ed3b0 0x55cfc0be1740 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 >> 172.21.15.146:0/642998785 conn(0x55cfc080a3c0 msgr2=0x55cfc0c714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 shutdown_connections 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 1 -- 172.21.15.146:0/642998785 wait complete. 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.715+0000 7f5e40b76740 20 client.4434 trim_cache size 0 max 16384 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "c6e9a678-3722-48f4-a297-70f509c13e29", 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4294, 2024-04-25T15:19:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-25T15:19:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 7.2544915960000003, 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 6, 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 25, 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 23, 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 13.98107119 2024-04-25T15:19:53.726 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e140744b0 msgr2=0x7f5e140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e140744b0 0x7f5e140769c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e341f9af0 tx=0x7f5e1c007de0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 msgr2=0x7f5e34153db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 0x7f5e34153db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400d5d0 tx=0x7f5e2400da90 comp rx=0 tx=0).stop 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 shutdown_connections 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e140744b0 0x7f5e140769c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e3415abc0 0x7f5e340cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e340d12a0 0x7f5e3415a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 --2- 172.21.15.146:0/2384233263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e340b45d0 0x7f5e34153db0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 >> 172.21.15.146:0/2384233263 conn(0x7f5e340d63b0 msgr2=0x7f5e3406d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 shutdown_connections 2024-04-25T15:19:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.716+0000 7f5e39cc6640 1 -- 172.21.15.146:0/2384233263 wait complete. 2024-04-25T15:19:53.729 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:19:53.729 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:53.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:19:53.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.836+0000 7fcc5d51f640 1 Processor -- start 2024-04-25T15:19:53.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.836+0000 7fcc5d51f640 1 -- start start 2024-04-25T15:19:53.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc5d51f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 0x7fcc5815ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc5d51f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcc580b45d0 0x7fcc580d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc5d51f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc580d1050 con 0x7fcc580d0c40 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc5d51f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcc580d4db0 con 0x7fcc580b45d0 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc5d51f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5806de00 con 0x7fcc580d12a0 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc577fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcc580d0c40 0x7fcc58163b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54142/0 (socket says 172.21.15.146:54142) 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc577fe640 1 -- 172.21.15.146:0/1478012641 learned_addr learned my addr 172.21.15.146:0/1478012641 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc567fc640 1 --2- 172.21.15.146:0/1478012641 >> v2:172.21.15.177:3300/0 conn(0x7fcc580b45d0 0x7fcc580d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc56ffd640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 0x7fcc5815ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005811623 0 0) 0x7fcc50001110 con 0x7fcc580d0c40 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc40003520 con 0x7fcc580d0c40 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3631928681 0 0) 0x7fcc40003520 con 0x7fcc580d0c40 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 msgr2=0x7fcc5815ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 0x7fcc5815ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 >> v2:172.21.15.177:3300/0 conn(0x7fcc580b45d0 msgr2=0x7fcc580d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 --2- 172.21.15.146:0/1478012641 >> v2:172.21.15.177:3300/0 conn(0x7fcc580b45d0 0x7fcc580d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc580d5000 con 0x7fcc580d0c40 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fcc50001840 con 0x7fcc580d0c40 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 >> v1:172.21.15.146:6789/0 conn(0x7fcc580d0c40 legacy=0x7fcc58163b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.838+0000 7fcc55ffb640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 0x7fcc400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc55ffb640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc40003f30 0x7fcc40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc55ffb640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc40004d40 0x7fcc40005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcc580d4db0 con 0x7fcc40004d40 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5806de00 con 0x7fcc40003f30 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcc4000bb50 con 0x7fcc580d12a0 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc577fe640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc40004d40 0x7fcc40005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc56ffd640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 0x7fcc400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc567fc640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc40003f30 0x7fcc40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc577fe640 1 -- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 msgr2=0x7fcc400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc577fe640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc580d12a0 0x7fcc400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc577fe640 1 -- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc40003f30 msgr2=0x7fcc40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc577fe640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc40003f30 0x7fcc40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.839+0000 7fcc577fe640 1 -- 172.21.15.146:0/1478012641 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc40003520 con 0x7fcc40004d40 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc577fe640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc40004d40 0x7fcc40005150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc50001050 tx=0x7fcc40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e95150e8106503 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000c0e0 con 0x7fcc40004d40 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc50034070 con 0x7fcc40004d40 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc55ffb640 1 -- 172.21.15.146:0/1478012641 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50005d10 con 0x7fcc40004d40 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc5d51f640 1 -- 172.21.15.146:0/1478012641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc40004d40 msgr2=0x7fcc40005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/1478012641 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc40004d40 0x7fcc40005150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc50001050 tx=0x7fcc40005830 comp rx=0 tx=0).stop 2024-04-25T15:19:53.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc577fe640 1 -- 172.21.15.146:0/1478012641 reap_dead start 2024-04-25T15:19:53.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc5d51f640 1 -- 172.21.15.146:0/1478012641 shutdown_connections 2024-04-25T15:19:53.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc5d51f640 1 -- 172.21.15.146:0/1478012641 >> 172.21.15.146:0/1478012641 conn(0x7fcc580d63b0 msgr2=0x7fcc580d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:53.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc5d51f640 1 -- 172.21.15.146:0/1478012641 shutdown_connections 2024-04-25T15:19:53.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.840+0000 7fcc5d51f640 1 -- 172.21.15.146:0/1478012641 wait complete. 2024-04-25T15:19:53.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.842+0000 7fcc5d51f640 1 Processor -- start 2024-04-25T15:19:53.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.842+0000 7fcc5d51f640 1 -- start start 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc5815b110 0x7fcc581f9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 0x7fcc581fa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc581e99b0 0x7fcc581e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5806c8d0 con 0x7fcc5815b520 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5806c5f0 con 0x7fcc5815b110 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcc580b8f10 con 0x7fcc581e99b0 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 0x7fcc581fa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 0x7fcc581fa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36714/0 (socket says 172.21.15.146:36714) 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 -- 172.21.15.146:0/3307768562 learned_addr learned my addr 172.21.15.146:0/3307768562 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc56ffd640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc5815b110 0x7fcc581f9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc577fe640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc581e99b0 0x7fcc581e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 -- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc581e99b0 msgr2=0x7fcc581e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc581e99b0 0x7fcc581e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 -- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc5815b110 msgr2=0x7fcc581f9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:53.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc5815b110 0x7fcc581f9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 -- 172.21.15.146:0/3307768562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc581ea300 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc567fc640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 0x7fcc581fa500 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c011960 tx=0x7fcc4c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c019070 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c004240 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc581ea590 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.844+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00f880 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.845+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc581eab50 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.845+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc5806a0e0 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.846+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcc4c01c070 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.846+0000 7fcc3f7fe640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcc440744b0 0x7fcc440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.847+0000 7fcc56ffd640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcc440744b0 0x7fcc440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.847+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcc4c095940 con 0x7fcc5815b520 2024-04-25T15:19:53.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.847+0000 7fcc56ffd640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcc440744b0 0x7fcc440769c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc580b4c30 tx=0x7fcc4800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:53.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:53.851+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc4c05f7d0 con 0x7fcc5815b520 2024-04-25T15:19:54.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.068+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcc580d3e30 con 0x7fcc5815b520 2024-04-25T15:19:54.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.068+0000 7fcc3f7fe640 1 -- 172.21.15.146:0/3307768562 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7fcc4c063ad0 con 0x7fcc5815b520 2024-04-25T15:19:54.069 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T15:19:54.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.070+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcc440744b0 msgr2=0x7fcc440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.070+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcc440744b0 0x7fcc440769c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc580b4c30 tx=0x7fcc4800b040 comp rx=0 tx=0).stop 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.070+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 msgr2=0x7fcc581fa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.070+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 0x7fcc581fa500 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c011960 tx=0x7fcc4c011e20 comp rx=0 tx=0).stop 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 shutdown_connections 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcc440744b0 0x7fcc440769c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc581e99b0 0x7fcc581e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc5815b520 0x7fcc581fa500 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 --2- 172.21.15.146:0/3307768562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc5815b110 0x7fcc581f9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 >> 172.21.15.146:0/3307768562 conn(0x7fcc580d63b0 msgr2=0x7fcc5815ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 shutdown_connections 2024-04-25T15:19:54.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54.071+0000 7fcc5d51f640 1 -- 172.21.15.146:0/3307768562 wait complete. 2024-04-25T15:19:54.082 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:19:54.083 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T15:19:54.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:54.083 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:19:54.098 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:19:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:19:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:19:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42767sec preferred_lft 42767sec 2024-04-25T15:19:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:19:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:19:54.100 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T15:19:54.101 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:19:54.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:19:54.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:54.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:54.280 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:19:54.303 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:19:54.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:54.305 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:19:54.369 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:19:54.370 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:19:54.370 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:19:54.370 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:19:54.370 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:19:54.370 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:19:54.371 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:19:54.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:19:54.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:19:54.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:19:54.371 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:19:54.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:19:54.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:54.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:54.676 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:19:54.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:54.692 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-04-25T15:19:54.747 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:19:54.747 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:19:54.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T15:19:54.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T15:19:54.747 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:19:54.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:19:54.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:54.863 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T15:19:54.863 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:19:54.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:19:54.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:54.992 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:19:54.993 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:19:54.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-25T15:19:54.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T15:19:54.993 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:19:55.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:19:55.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:55.067 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T15:19:55.067 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:19:55.067 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:55.092 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T15:19:55.092 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:19:55.092 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:55.146 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:19:55.211 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:19:55.266 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:19:55.267 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T15:19:55.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:55.268 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:19:55.321 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:19:55.321 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T15:19:55.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:19:55.390 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:19:55.446 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:19:55.446 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:19:55.446 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:19:55.446 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:19:55.447 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:19:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T15:19:55.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:55.449 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:19:55.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi146.stderr:2024-04-25T15:19:55.470+0000 7fee5f02f480 -1 init, newargv = 0x5588f9b25140 newargc=15 2024-04-25T15:19:55.471 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi146.stderr:ceph-fuse[41860]: starting ceph client 2024-04-25T15:19:55.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi146.stderr:ceph-fuse[41860]: starting fuse 2024-04-25T15:19:55.486 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:19:55.486 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:19:55.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:19:55.486 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:19:55.487 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.admin")) 2024-04-25T15:19:55.488 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:19:55.603 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.admin.41860.asok 2024-04-25T15:19:55.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:55.613 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.41860.asok 2024-04-25T15:19:55.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:55.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.41860.asok status 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "admin", 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "41860", 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-04-25T15:19:55.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4446, 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4446 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 233028837 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:19:55.757 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 233028837 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4446 192.168.0.1:0/233028837", 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/233028837", 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 6, 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 25, 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:19:55.758 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:19:55.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:55.764 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-04-25T15:19:55.764 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:55.779 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-04-25T15:19:55.779 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:55.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:55.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:55.845 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T15:19:55.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:19:55.950 INFO:teuthology.orchestra.run.smithi146.stdout:# file: . 2024-04-25T15:19:55.950 INFO:teuthology.orchestra.run.smithi146.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T15:19:55.950 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:55.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:19:55.957 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:55.957 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:19:55.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:56.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.119+0000 7fca7ec82640 1 Processor -- start 2024-04-25T15:19:56.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.119+0000 7fca7ec82640 1 -- start start 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7ec82640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 0x7fca78153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7ec82640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca780cf420 con 0x7fca780d0ac0 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7ec82640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca780cc270 con 0x7fca780d01a0 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7ec82640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca781541e0 con 0x7fca780cbc90 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7d21f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fca780d0ac0 0x7fca780d0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54150/0 (socket says 172.21.15.146:54150) 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7d21f640 1 -- 172.21.15.146:0/2128841148 learned_addr learned my addr 172.21.15.146:0/2128841148 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.120+0000 7fca7ca1e640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 0x7fca78153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 941797910 0 0) 0x7fca780cf420 con 0x7fca780d0ac0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca60003520 con 0x7fca780d0ac0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984383886 0 0) 0x7fca780cc270 con 0x7fca780d01a0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca780cf420 con 0x7fca780d01a0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3204078888 0 0) 0x7fca60003520 con 0x7fca780d0ac0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 msgr2=0x7fca78153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 0x7fca78153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 >> v1:172.21.15.177:6789/0 conn(0x7fca780d01a0 legacy=0x7fca7815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca780cf6c0 con 0x7fca780d0ac0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fca70001a10 con 0x7fca780d0ac0 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 >> v1:172.21.15.146:6789/0 conn(0x7fca780d0ac0 legacy=0x7fca780d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 0x7fca60003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca600040d0 0x7fca60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca60004d40 0x7fca60005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fca6000baf0 con 0x7fca60004d40 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca6000bca0 con 0x7fca600040d0 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca6000bea0 con 0x7fca780cbc90 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca60004d40 0x7fca60005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7ca1e640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 0x7fca60003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca77fff640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca600040d0 0x7fca60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7d21f640 1 -- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 msgr2=0x7fca60003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca780cbc90 0x7fca60003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7d21f640 1 -- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca600040d0 msgr2=0x7fca60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca600040d0 0x7fca60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.121+0000 7fca7d21f640 1 -- 172.21.15.146:0/2128841148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca60003520 con 0x7fca60004d40 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.122+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca60004d40 0x7fca60005150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fca781578e0 tx=0x7fca60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd53de2944f7c42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.122+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7000a430 con 0x7fca60004d40 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.122+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca7002f070 con 0x7fca60004d40 2024-04-25T15:19:56.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.122+0000 7fca777fe640 1 -- 172.21.15.146:0/2128841148 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7002bba0 con 0x7fca60004d40 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7ec82640 1 -- 172.21.15.146:0/2128841148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca60004d40 msgr2=0x7fca60005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2128841148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca60004d40 0x7fca60005150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fca781578e0 tx=0x7fca60005830 comp rx=0 tx=0).stop 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7d21f640 1 -- 172.21.15.146:0/2128841148 reap_dead start 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7ec82640 1 -- 172.21.15.146:0/2128841148 shutdown_connections 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7ec82640 1 -- 172.21.15.146:0/2128841148 >> 172.21.15.146:0/2128841148 conn(0x7fca780ce0c0 msgr2=0x7fca780ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7ec82640 1 -- 172.21.15.146:0/2128841148 shutdown_connections 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.123+0000 7fca7ec82640 1 -- 172.21.15.146:0/2128841148 wait complete. 2024-04-25T15:19:56.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.125+0000 7fca7ec82640 1 Processor -- start 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.125+0000 7fca7ec82640 1 -- start start 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ec82640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca78156930 0x7fca7814d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ec82640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78153f50 0x7fca78152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ec82640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 0x7fca78152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ec82640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fca780d05b0 con 0x7fca78154360 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ec82640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca780cfd10 con 0x7fca78153f50 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ec82640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca780d0730 con 0x7fca78156930 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 0x7fca78152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 0x7fca78152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36730/0 (socket says 172.21.15.146:36730) 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 -- 172.21.15.146:0/2726806987 learned_addr learned my addr 172.21.15.146:0/2726806987 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca77fff640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78153f50 0x7fca78152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7ca1e640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca78156930 0x7fca7814d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 -- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca78156930 msgr2=0x7fca7814d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca78156930 0x7fca7814d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 -- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78153f50 msgr2=0x7fca78152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78153f50 0x7fca78152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.126+0000 7fca7d21f640 1 -- 172.21.15.146:0/2726806987 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca781f9ba0 con 0x7fca78154360 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.127+0000 7fca7d21f640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 0x7fca78152ca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca7000a890 tx=0x7fca780cf890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.127+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca70044070 con 0x7fca78154360 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.127+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca7002ce10 con 0x7fca78154360 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.127+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7000a430 con 0x7fca78154360 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.127+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca781f9e80 con 0x7fca78154360 2024-04-25T15:19:56.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.127+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca781fa440 con 0x7fca78154360 2024-04-25T15:19:56.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.128+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca7806a0e0 con 0x7fca78154360 2024-04-25T15:19:56.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.129+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fca70047070 con 0x7fca78154360 2024-04-25T15:19:56.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.129+0000 7fca757fa640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca640744b0 0x7fca640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.129+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fca700b5360 con 0x7fca78154360 2024-04-25T15:19:56.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.129+0000 7fca7ca1e640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca640744b0 0x7fca640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.129+0000 7fca7ca1e640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca640744b0 0x7fca640769c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca780ccdc0 tx=0x7fca68009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.134+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca7007f990 con 0x7fca78154360 2024-04-25T15:19:56.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.310+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca780711c0 con 0x7fca78154360 2024-04-25T15:19:56.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.312+0000 7fca757fa640 1 -- 172.21.15.146:0/2726806987 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7fca7007f290 con 0x7fca78154360 2024-04-25T15:19:56.313 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:56.313 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca640744b0 msgr2=0x7fca640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca640744b0 0x7fca640769c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca780ccdc0 tx=0x7fca68009190 comp rx=0 tx=0).stop 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 msgr2=0x7fca78152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 0x7fca78152ca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca7000a890 tx=0x7fca780cf890 comp rx=0 tx=0).stop 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 shutdown_connections 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca640744b0 0x7fca640769c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca78154360 0x7fca78152ca0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78153f50 0x7fca78152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 --2- 172.21.15.146:0/2726806987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca78156930 0x7fca7814d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 >> 172.21.15.146:0/2726806987 conn(0x7fca780ce0c0 msgr2=0x7fca780b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 shutdown_connections 2024-04-25T15:19:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.314+0000 7fca7ec82640 1 -- 172.21.15.146:0/2726806987 wait complete. 2024-04-25T15:19:56.326 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:19:56.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:19:56.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.437+0000 7f97763b2640 1 Processor -- start 2024-04-25T15:19:56.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.437+0000 7f97763b2640 1 -- start start 2024-04-25T15:19:56.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f97763b2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9770071d30 0x7f97700d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f97763b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f97700d3300 0x7f97700cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f97763b2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97700d2260 con 0x7f97700b2cd0 2024-04-25T15:19:56.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f97763b2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f97700d2540 con 0x7f97700d3300 2024-04-25T15:19:56.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f97763b2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f97700d2820 con 0x7f9770071d30 2024-04-25T15:19:56.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f977494f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f97700d3300 0x7f97700cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f976ffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f97700b2cd0 0x7f9770071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54156/0 (socket says 172.21.15.146:54156) 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f976ffff640 1 -- 172.21.15.146:0/3708776360 learned_addr learned my addr 172.21.15.146:0/3708776360 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f976f7fe640 1 --2- 172.21.15.146:0/3708776360 >> v2:172.21.15.177:3300/0 conn(0x7f9770071d30 0x7f97700d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f977494f640 1 -- 172.21.15.146:0/3708776360 >> v2:172.21.15.177:3300/0 conn(0x7f9770071d30 msgr2=0x7f97700d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f977494f640 1 --2- 172.21.15.146:0/3708776360 >> v2:172.21.15.177:3300/0 conn(0x7f9770071d30 0x7f97700d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f977494f640 1 -- 172.21.15.146:0/3708776360 >> v1:172.21.15.146:6789/0 conn(0x7f97700b2cd0 legacy=0x7f9770071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f977494f640 1 -- 172.21.15.146:0/3708776360 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97700d2b00 con 0x7f97700d3300 2024-04-25T15:19:56.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.439+0000 7f976effd640 1 -- 172.21.15.146:0/3708776360 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367466681 0 0) 0x7f97700d2260 con 0x7f97700b2cd0 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.440+0000 7f977494f640 1 --2- 172.21.15.146:0/3708776360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f97700cde40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9770075280 tx=0x7f976802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24a256a1a97666c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.440+0000 7f976effd640 1 -- 172.21.15.146:0/3708776360 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768007740 con 0x7f97700d3300 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.440+0000 7f976effd640 1 -- 172.21.15.146:0/3708776360 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97680078e0 con 0x7f97700d3300 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.440+0000 7f976effd640 1 -- 172.21.15.146:0/3708776360 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768007cb0 con 0x7f97700d3300 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 -- 172.21.15.146:0/3708776360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 msgr2=0x7f97700cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 --2- 172.21.15.146:0/3708776360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f97700cde40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9770075280 tx=0x7f976802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 -- 172.21.15.146:0/3708776360 shutdown_connections 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 --2- 172.21.15.146:0/3708776360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f97700cde40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 --2- 172.21.15.146:0/3708776360 >> v2:172.21.15.177:3300/0 conn(0x7f9770071d30 0x7f97700d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 -- 172.21.15.146:0/3708776360 >> 172.21.15.146:0/3708776360 conn(0x7f97700b52e0 msgr2=0x7f97700b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:56.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 -- 172.21.15.146:0/3708776360 shutdown_connections 2024-04-25T15:19:56.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.441+0000 7f97763b2640 1 -- 172.21.15.146:0/3708776360 wait complete. 2024-04-25T15:19:56.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.443+0000 7f97763b2640 1 Processor -- start 2024-04-25T15:19:56.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.443+0000 7f97763b2640 1 -- start start 2024-04-25T15:19:56.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f97763b2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97700b2cd0 0x7f9770154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f97763b2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f9770155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f97763b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97701563a0 0x7f97700b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f97763b2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f977008dbf0 con 0x7f97700d3300 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f97763b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97700cfb70 con 0x7f97701563a0 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f97763b2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f977008dd70 con 0x7f97700b2cd0 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f976f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f9770155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f976f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f9770155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36752/0 (socket says 172.21.15.146:36752) 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f976f7fe640 1 -- 172.21.15.146:0/242757379 learned_addr learned my addr 172.21.15.146:0/242757379 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f976ffff640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97700b2cd0 0x7f9770154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.444+0000 7f977494f640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97701563a0 0x7f97700b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976f7fe640 1 -- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97700b2cd0 msgr2=0x7f9770154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976f7fe640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97700b2cd0 0x7f9770154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976f7fe640 1 -- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97701563a0 msgr2=0x7f97700b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976f7fe640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97701563a0 0x7f97700b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976f7fe640 1 -- 172.21.15.146:0/242757379 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97701e6190 con 0x7f97700d3300 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976f7fe640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f9770155e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f976400e870 tx=0x7f976400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764017070 con 0x7f97700d3300 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f976400c8e0 con 0x7f97700d3300 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97701e6470 con 0x7f97700d3300 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976400ccd0 con 0x7f97700d3300 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.445+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97701e6a30 con 0x7f97700d3300 2024-04-25T15:19:56.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.446+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977006a0e0 con 0x7f97700d3300 2024-04-25T15:19:56.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.447+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f976401a070 con 0x7f97700d3300 2024-04-25T15:19:56.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.447+0000 7f976cff9640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f975c0744e0 0x7f975c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.447+0000 7f976ffff640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f975c0744e0 0x7f975c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.447+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f97640955c0 con 0x7f97700d3300 2024-04-25T15:19:56.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.448+0000 7f976ffff640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f975c0744e0 0x7f975c0769f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97600010e0 tx=0x7f9760009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.452+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f976405f4e0 con 0x7f97700d3300 2024-04-25T15:19:56.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.629+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97700ce1a0 con 0x7f97700d3300 2024-04-25T15:19:56.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.629+0000 7f976cff9640 1 -- 172.21.15.146:0/242757379 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f97640637e0 con 0x7f97700d3300 2024-04-25T15:19:56.630 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:19:56.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f975c0744e0 msgr2=0x7f975c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f975c0744e0 0x7f975c0769f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97600010e0 tx=0x7f9760009040 comp rx=0 tx=0).stop 2024-04-25T15:19:56.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 msgr2=0x7f9770155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f9770155e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f976400e870 tx=0x7f976400ed30 comp rx=0 tx=0).stop 2024-04-25T15:19:56.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 shutdown_connections 2024-04-25T15:19:56.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f975c0744e0 0x7f975c0769f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97701563a0 0x7f97700b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97700d3300 0x7f9770155e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 --2- 172.21.15.146:0/242757379 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97700b2cd0 0x7f9770154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 >> 172.21.15.146:0/242757379 conn(0x7f97700b52e0 msgr2=0x7f9770159140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:56.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 shutdown_connections 2024-04-25T15:19:56.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.631+0000 7f97763b2640 1 -- 172.21.15.146:0/242757379 wait complete. 2024-04-25T15:19:56.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:19:56.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.753+0000 7fef8c92d640 1 Processor -- start 2024-04-25T15:19:56.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.753+0000 7fef8c92d640 1 -- start start 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8c92d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fef880ce2a0 0x7fef88160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8c92d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef8806f7b0 con 0x7fef8806f2a0 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8c92d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef8809b700 con 0x7fef88160b00 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8c92d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fef880b1f30 con 0x7fef880ce2a0 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fef8806f2a0 0x7fef880cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54172/0 (socket says 172.21.15.146:54172) 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8659c640 1 -- 172.21.15.146:0/207081239 learned_addr learned my addr 172.21.15.146:0/207081239 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:56.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.754+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415916181 0 0) 0x7fef8806f7b0 con 0x7fef8806f2a0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef70003520 con 0x7fef8806f2a0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 --2- 172.21.15.146:0/207081239 >> v2:172.21.15.177:3300/0 conn(0x7fef880ce2a0 0x7fef88160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 350968124 0 0) 0x7fef70003520 con 0x7fef8806f2a0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 >> v2:172.21.15.177:3300/0 conn(0x7fef880ce2a0 msgr2=0x7fef88160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 --2- 172.21.15.146:0/207081239 >> v2:172.21.15.177:3300/0 conn(0x7fef880ce2a0 0x7fef88160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 >> v1:172.21.15.177:6789/0 conn(0x7fef88160b00 legacy=0x7fef8815c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8809b9e0 con 0x7fef8806f2a0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fef780019f0 con 0x7fef8806f2a0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 >> v1:172.21.15.146:6789/0 conn(0x7fef8806f2a0 legacy=0x7fef880cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef700039b0 0x7fef70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef70004800 0x7fef70004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fef880b1f30 con 0x7fef700051c0 2024-04-25T15:19:56.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef7000c160 con 0x7fef70004800 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fef7000c360 con 0x7fef700039b0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef8659c640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef700039b0 0x7fef70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef85d9b640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef70004800 0x7fef70004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 -- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef700039b0 msgr2=0x7fef70003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef700039b0 0x7fef70003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 -- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef70004800 msgr2=0x7fef70004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef70004800 0x7fef70004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.755+0000 7fef86d9d640 1 -- 172.21.15.146:0/207081239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef70003520 con 0x7fef700051c0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.756+0000 7fef86d9d640 1 -- 172.21.15.146:0/207081239 reap_dead start 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.756+0000 7fef86d9d640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef7000b9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fef80009ac0 tx=0x7fef70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dafaa88d90dc37b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.756+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80009690 con 0x7fef700051c0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.756+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef80004020 con 0x7fef700051c0 2024-04-25T15:19:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.756+0000 7fef8559a640 1 -- 172.21.15.146:0/207081239 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80004410 con 0x7fef700051c0 2024-04-25T15:19:56.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 -- 172.21.15.146:0/207081239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 msgr2=0x7fef7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef7000b9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fef80009ac0 tx=0x7fef70005830 comp rx=0 tx=0).stop 2024-04-25T15:19:56.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 -- 172.21.15.146:0/207081239 shutdown_connections 2024-04-25T15:19:56.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 --2- 172.21.15.146:0/207081239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef7000b9c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 -- 172.21.15.146:0/207081239 >> 172.21.15.146:0/207081239 conn(0x7fef880b7490 msgr2=0x7fef880b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 -- 172.21.15.146:0/207081239 shutdown_connections 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.757+0000 7fef8c92d640 1 -- 172.21.15.146:0/207081239 wait complete. 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.759+0000 7fef8c92d640 1 Processor -- start 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.759+0000 7fef8c92d640 1 -- start start 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8c92d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef880d2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8c92d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef8815ab50 0x7fef8815af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8c92d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef8815d540 0x7fef8815d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8c92d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fef88161090 con 0x7fef700051c0 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8c92d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef88160f10 con 0x7fef8815ab50 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8c92d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fef88161210 con 0x7fef8815d540 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef880d2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef880d2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36764/0 (socket says 172.21.15.146:36764) 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.760+0000 7fef8659c640 1 -- 172.21.15.146:0/1916733310 learned_addr learned my addr 172.21.15.146:0/1916733310 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef86d9d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef8815d540 0x7fef8815d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef85d9b640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef8815ab50 0x7fef8815af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8659c640 1 -- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef8815d540 msgr2=0x7fef8815d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8659c640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef8815d540 0x7fef8815d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8659c640 1 -- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef8815ab50 msgr2=0x7fef8815af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8659c640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef8815ab50 0x7fef8815af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8659c640 1 -- 172.21.15.146:0/1916733310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef880d16f0 con 0x7fef700051c0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef85d9b640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef8815ab50 0x7fef8815af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8659c640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef880d2790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef78004fa0 tx=0x7fef7800a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7800ae10 con 0x7fef700051c0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78016070 con 0x7fef700051c0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7801b6b0 con 0x7fef700051c0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef880d19d0 con 0x7fef700051c0 2024-04-25T15:19:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.761+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef880d1f90 con 0x7fef700051c0 2024-04-25T15:19:56.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.762+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8806a0e0 con 0x7fef700051c0 2024-04-25T15:19:56.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.763+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fef78012450 con 0x7fef700051c0 2024-04-25T15:19:56.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.763+0000 7fef76ffd640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fef680744b0 0x7fef680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:56.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.763+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef78097e90 con 0x7fef700051c0 2024-04-25T15:19:56.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.763+0000 7fef85d9b640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fef680744b0 0x7fef680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:56.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.764+0000 7fef85d9b640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fef680744b0 0x7fef680769c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef881f73e0 tx=0x7fef7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.768+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef780622e0 con 0x7fef700051c0 2024-04-25T15:19:56.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.943+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fef880c5f30 con 0x7fef700051c0 2024-04-25T15:19:56.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.944+0000 7fef76ffd640 1 -- 172.21.15.146:0/1916733310 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fef78061be0 con 0x7fef700051c0 2024-04-25T15:19:56.945 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fef680744b0 msgr2=0x7fef680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fef680744b0 0x7fef680769c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef881f73e0 tx=0x7fef7c009040 comp rx=0 tx=0).stop 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 msgr2=0x7fef880d2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef880d2790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef78004fa0 tx=0x7fef7800a730 comp rx=0 tx=0).stop 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 shutdown_connections 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fef680744b0 0x7fef680769c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef8815d540 0x7fef8815d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef8815ab50 0x7fef8815af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 --2- 172.21.15.146:0/1916733310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef700051c0 0x7fef880d2790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:56.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 >> 172.21.15.146:0/1916733310 conn(0x7fef880b7490 msgr2=0x7fef88071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:56.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 shutdown_connections 2024-04-25T15:19:56.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:56.946+0000 7fef8c92d640 1 -- 172.21.15.146:0/1916733310 wait complete. 2024-04-25T15:19:56.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:19:57.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.068+0000 7fe57c6d8640 1 Processor -- start 2024-04-25T15:19:57.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.068+0000 7fe57c6d8640 1 -- start start 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57c6d8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe5740b7430 0x7fe5740b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57c6d8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5740b2fe0 con 0x7fe57406c550 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57c6d8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5740d1ff0 con 0x7fe5740d2be0 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57c6d8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe5740b3820 con 0x7fe5740b7430 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe579c73640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe57406c550 0x7fe5740d24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54180/0 (socket says 172.21.15.146:54180) 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe579c73640 1 -- 172.21.15.146:0/872792858 learned_addr learned my addr 172.21.15.146:0/872792858 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57a474640 1 --2- 172.21.15.146:0/872792858 >> v2:172.21.15.177:3300/0 conn(0x7fe5740b7430 0x7fe5740b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe579472640 1 -- 172.21.15.146:0/872792858 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2115182238 0 0) 0x7fe5740b2fe0 con 0x7fe57406c550 2024-04-25T15:19:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe579472640 1 -- 172.21.15.146:0/872792858 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe560003520 con 0x7fe57406c550 2024-04-25T15:19:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe579472640 1 -- 172.21.15.146:0/872792858 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2614299559 0 0) 0x7fe560003520 con 0x7fe57406c550 2024-04-25T15:19:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57a474640 1 -- 172.21.15.146:0/872792858 >> v1:172.21.15.177:6789/0 conn(0x7fe5740d2be0 legacy=0x7fe5740d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57a474640 1 -- 172.21.15.146:0/872792858 >> v1:172.21.15.146:6789/0 conn(0x7fe57406c550 legacy=0x7fe5740d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.070+0000 7fe57a474640 1 -- 172.21.15.146:0/872792858 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5740b3280 con 0x7fe5740b7430 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.071+0000 7fe57a474640 1 --2- 172.21.15.146:0/872792858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740b7430 0x7fe5740b2310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe5700097a0 tx=0x7fe57002a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d0b458789d8cbe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.071+0000 7fe579472640 1 -- 172.21.15.146:0/872792858 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570037070 con 0x7fe5740b7430 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.071+0000 7fe579472640 1 -- 172.21.15.146:0/872792858 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57002abc0 con 0x7fe5740b7430 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.071+0000 7fe579472640 1 -- 172.21.15.146:0/872792858 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564001690 con 0x7fe5740b7430 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 -- 172.21.15.146:0/872792858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740b7430 msgr2=0x7fe5740b2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/872792858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740b7430 0x7fe5740b2310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe5700097a0 tx=0x7fe57002a130 comp rx=0 tx=0).stop 2024-04-25T15:19:57.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 -- 172.21.15.146:0/872792858 shutdown_connections 2024-04-25T15:19:57.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/872792858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740b7430 0x7fe5740b2310 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 -- 172.21.15.146:0/872792858 >> 172.21.15.146:0/872792858 conn(0x7fe57409b2b0 msgr2=0x7fe5740b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:57.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 -- 172.21.15.146:0/872792858 shutdown_connections 2024-04-25T15:19:57.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.072+0000 7fe57c6d8640 1 -- 172.21.15.146:0/872792858 wait complete. 2024-04-25T15:19:57.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.074+0000 7fe57c6d8640 1 Processor -- start 2024-04-25T15:19:57.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.074+0000 7fe57c6d8640 1 -- start start 2024-04-25T15:19:57.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57c6d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe57406c550 0x7fe574151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57c6d8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 0x7fe5740af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57c6d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740d2be0 0x7fe5741f9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57c6d8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe574160280 con 0x7fe5740b7430 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57c6d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740b2850 con 0x7fe5740d2be0 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57c6d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe574160400 con 0x7fe57406c550 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 0x7fe5740af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 0x7fe5740af3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36770/0 (socket says 172.21.15.146:36770) 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 -- 172.21.15.146:0/2506979368 learned_addr learned my addr 172.21.15.146:0/2506979368 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57ac75640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740d2be0 0x7fe5741f9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe57a474640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe57406c550 0x7fe574151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 -- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe57406c550 msgr2=0x7fe574151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe57406c550 0x7fe574151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 -- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740d2be0 msgr2=0x7fe5741f9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.075+0000 7fe579c73640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740d2be0 0x7fe5741f9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe579c73640 1 -- 172.21.15.146:0/2506979368 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5740b0d70 con 0x7fe5740b7430 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe57a474640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe57406c550 0x7fe574151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe579c73640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 0x7fe5740af3f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe564001200 tx=0x7fe56400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564018070 con 0x7fe5740b7430 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe564007500 con 0x7fe5740b7430 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5740b0160 con 0x7fe5740b7430 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564004a20 con 0x7fe5740b7430 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.076+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5740b0720 con 0x7fe5740b7430 2024-04-25T15:19:57.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.077+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5740afed0 con 0x7fe5740b7430 2024-04-25T15:19:57.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.077+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe564004bc0 con 0x7fe5740b7430 2024-04-25T15:19:57.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.078+0000 7fe56affd640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe5580744b0 0x7fe5580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.078+0000 7fe57a474640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe5580744b0 0x7fe5580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.078+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe5640948d0 con 0x7fe5740b7430 2024-04-25T15:19:57.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.078+0000 7fe57a474640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe5580744b0 0x7fe5580769c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe570034000 tx=0x7fe57002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.082+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe56405eca0 con 0x7fe5740b7430 2024-04-25T15:19:57.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.257+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe57406a0e0 con 0x7fe5740b7430 2024-04-25T15:19:57.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.257+0000 7fe56affd640 1 -- 172.21.15.146:0/2506979368 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fe56405e5a0 con 0x7fe5740b7430 2024-04-25T15:19:57.258 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:19:57.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe5580744b0 msgr2=0x7fe5580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe5580744b0 0x7fe5580769c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe570034000 tx=0x7fe57002e000 comp rx=0 tx=0).stop 2024-04-25T15:19:57.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 msgr2=0x7fe5740af3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 0x7fe5740af3f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe564001200 tx=0x7fe56400c9b0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 shutdown_connections 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe5580744b0 0x7fe5580769c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5740d2be0 0x7fe5741f9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5740b7430 0x7fe5740af3f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 --2- 172.21.15.146:0/2506979368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe57406c550 0x7fe574151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 >> 172.21.15.146:0/2506979368 conn(0x7fe57409b2b0 msgr2=0x7fe5740cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 shutdown_connections 2024-04-25T15:19:57.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.259+0000 7fe57c6d8640 1 -- 172.21.15.146:0/2506979368 wait complete. 2024-04-25T15:19:57.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:19:57.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.382+0000 7f2b53861640 1 Processor -- start 2024-04-25T15:19:57.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.382+0000 7f2b53861640 1 -- start start 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.383+0000 7f2b53861640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.383+0000 7f2b53861640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2b4c06cde0 0x7f2b4c06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.383+0000 7f2b53861640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b4c0b37a0 con 0x7f2b4c0b3390 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.383+0000 7f2b53861640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4c06d730 con 0x7f2b4c06cde0 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.383+0000 7f2b53861640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c06e4a0 con 0x7f2b4c0cdaa0 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2b4c06cde0 0x7f2b4c06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b50dfc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2b4c06cde0 0x7f2b4c06d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46386/0 (socket says 172.21.15.146:46386) 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 -- 172.21.15.146:0/4213175616 learned_addr learned my addr 172.21.15.146:0/4213175616 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b515fd640 1 --1- 172.21.15.146:0/4213175616 >> v1:172.21.15.177:6789/0 conn(0x7f2b4c0b3390 0x7f2b4c0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38100/0 (socket says 172.21.15.146:38100) 2024-04-25T15:19:57.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 -- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 msgr2=0x7f2b4c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:57.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 --2- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 -- 172.21.15.146:0/4213175616 >> v1:172.21.15.177:6789/0 conn(0x7f2b4c0b3390 legacy=0x7f2b4c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b51dfe640 1 -- 172.21.15.146:0/4213175616 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c0ce3f0 con 0x7f2b4c06cde0 2024-04-25T15:19:57.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.384+0000 7f2b50dfc640 1 --2- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:19:57.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.385+0000 7f2b51dfe640 1 --2- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c06d1f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400af60 tx=0x7f2b4402b830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a6b269777bb183b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.385+0000 7f2b3ffff640 1 -- 172.21.15.146:0/4213175616 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44038070 con 0x7f2b4c06cde0 2024-04-25T15:19:57.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.385+0000 7f2b3ffff640 1 -- 172.21.15.146:0/4213175616 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b4400a2d0 con 0x7f2b4c06cde0 2024-04-25T15:19:57.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.385+0000 7f2b3ffff640 1 -- 172.21.15.146:0/4213175616 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44008720 con 0x7f2b4c06cde0 2024-04-25T15:19:57.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 -- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c06cde0 msgr2=0x7f2b4c06d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 --2- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c06d1f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400af60 tx=0x7f2b4402b830 comp rx=0 tx=0).stop 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 -- 172.21.15.146:0/4213175616 shutdown_connections 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 --2- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c06d1f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 --2- 172.21.15.146:0/4213175616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 -- 172.21.15.146:0/4213175616 >> 172.21.15.146:0/4213175616 conn(0x7f2b4c09b2b0 msgr2=0x7f2b4c0b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 -- 172.21.15.146:0/4213175616 shutdown_connections 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.386+0000 7f2b53861640 1 -- 172.21.15.146:0/4213175616 wait complete. 2024-04-25T15:19:57.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.388+0000 7f2b53861640 1 Processor -- start 2024-04-25T15:19:57.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.388+0000 7f2b53861640 1 -- start start 2024-04-25T15:19:57.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b53861640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b53861640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c0b3390 0x7f2b4c0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b53861640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b53861640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0cd580 con 0x7f2b4c06cde0 2024-04-25T15:19:57.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b53861640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0cd2a0 con 0x7f2b4c0b3390 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b53861640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c06e740 con 0x7f2b4c0cdaa0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b515fd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b515fd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36776/0 (socket says 172.21.15.146:36776) 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b515fd640 1 -- 172.21.15.146:0/1079540749 learned_addr learned my addr 172.21.15.146:0/1079540749 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b51dfe640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.389+0000 7f2b50dfc640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c0b3390 0x7f2b4c0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b515fd640 1 -- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 msgr2=0x7f2b4c0ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b515fd640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b515fd640 1 -- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c0b3390 msgr2=0x7f2b4c0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b515fd640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c0b3390 0x7f2b4c0cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b515fd640 1 -- 172.21.15.146:0/1079540749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c1f23a0 con 0x7f2b4c06cde0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b515fd640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c159540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800c3f0 tx=0x7f2b4800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b480074e0 con 0x7f2b4c06cde0 2024-04-25T15:19:57.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b48007680 con 0x7f2b4c06cde0 2024-04-25T15:19:57.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b48004a00 con 0x7f2b4c06cde0 2024-04-25T15:19:57.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c1f2680 con 0x7f2b4c06cde0 2024-04-25T15:19:57.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.390+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c1f2c40 con 0x7f2b4c06cde0 2024-04-25T15:19:57.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.391+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c06a0e0 con 0x7f2b4c06cde0 2024-04-25T15:19:57.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.392+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2b4801a020 con 0x7f2b4c06cde0 2024-04-25T15:19:57.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.392+0000 7f2b3dffb640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2b300744e0 0x7f2b300769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.392+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2b48094b10 con 0x7f2b4c06cde0 2024-04-25T15:19:57.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.392+0000 7f2b50dfc640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2b300744e0 0x7f2b300769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.393+0000 7f2b50dfc640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2b300744e0 0x7f2b300769f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c15b4c0 tx=0x7f2b40009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.397+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b4805e910 con 0x7f2b4c06cde0 2024-04-25T15:19:57.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.572+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2b4c0b76e0 con 0x7f2b4c06cde0 2024-04-25T15:19:57.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.573+0000 7f2b3dffb640 1 -- 172.21.15.146:0/1079540749 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f2b48062c10 con 0x7f2b4c06cde0 2024-04-25T15:19:57.574 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2b300744e0 msgr2=0x7f2b300769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2b300744e0 0x7f2b300769f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c15b4c0 tx=0x7f2b40009190 comp rx=0 tx=0).stop 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 msgr2=0x7f2b4c159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c159540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800c3f0 tx=0x7f2b4800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 shutdown_connections 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2b300744e0 0x7f2b300769f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b4c0cdaa0 0x7f2b4c0ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b4c0b3390 0x7f2b4c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 --2- 172.21.15.146:0/1079540749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b4c06cde0 0x7f2b4c159540 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 >> 172.21.15.146:0/1079540749 conn(0x7f2b4c09b2b0 msgr2=0x7f2b4c0b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 shutdown_connections 2024-04-25T15:19:57.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.575+0000 7f2b53861640 1 -- 172.21.15.146:0/1079540749 wait complete. 2024-04-25T15:19:57.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:19:57.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.695+0000 7ff67c9b7640 1 Processor -- start 2024-04-25T15:19:57.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.695+0000 7ff67c9b7640 1 -- start start 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.696+0000 7ff67c9b7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff6780b5950 0x7ff6780d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.696+0000 7ff67c9b7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff67815ffa0 0x7ff6780b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.696+0000 7ff67c9b7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6780d2230 con 0x7ff6780b6610 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.696+0000 7ff67c9b7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff6780b2f50 con 0x7ff6780b5950 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.696+0000 7ff67c9b7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff6780b2ac0 con 0x7ff67815ffa0 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff67659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff6780b6610 0x7ff6780b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54190/0 (socket says 172.21.15.146:54190) 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff67659c640 1 -- 172.21.15.146:0/3215353201 learned_addr learned my addr 172.21.15.146:0/3215353201 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff675d9b640 1 --2- 172.21.15.146:0/3215353201 >> v2:172.21.15.146:3300/0 conn(0x7ff6780b5950 0x7ff6780d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff676d9d640 1 --2- 172.21.15.146:0/3215353201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff67815ffa0 0x7ff6780b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff675d9b640 1 -- 172.21.15.146:0/3215353201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff67815ffa0 msgr2=0x7ff6780b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff675d9b640 1 --2- 172.21.15.146:0/3215353201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff67815ffa0 0x7ff6780b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff67559a640 1 -- 172.21.15.146:0/3215353201 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1570897914 0 0) 0x7ff6780d2230 con 0x7ff6780b6610 2024-04-25T15:19:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff675d9b640 1 -- 172.21.15.146:0/3215353201 >> v1:172.21.15.146:6789/0 conn(0x7ff6780b6610 legacy=0x7ff6780b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.697+0000 7ff675d9b640 1 -- 172.21.15.146:0/3215353201 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6780cfa80 con 0x7ff6780b5950 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.698+0000 7ff675d9b640 1 --2- 172.21.15.146:0/3215353201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b5950 0x7ff6780d2df0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0097a0 tx=0x7ff66c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8109516314fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.698+0000 7ff67559a640 1 -- 172.21.15.146:0/3215353201 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c037070 con 0x7ff6780b5950 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.698+0000 7ff67559a640 1 -- 172.21.15.146:0/3215353201 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66c032440 con 0x7ff6780b5950 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.698+0000 7ff67559a640 1 -- 172.21.15.146:0/3215353201 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c032810 con 0x7ff6780b5950 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.698+0000 7ff67c9b7640 1 -- 172.21.15.146:0/3215353201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b5950 msgr2=0x7ff6780d2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.698+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/3215353201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b5950 0x7ff6780d2df0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0097a0 tx=0x7ff66c02a180 comp rx=0 tx=0).stop 2024-04-25T15:19:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.699+0000 7ff67c9b7640 1 -- 172.21.15.146:0/3215353201 shutdown_connections 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.699+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/3215353201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff67815ffa0 0x7ff6780b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.699+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/3215353201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6780b5950 0x7ff6780d2df0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.699+0000 7ff67c9b7640 1 -- 172.21.15.146:0/3215353201 >> 172.21.15.146:0/3215353201 conn(0x7ff6780d0240 msgr2=0x7ff6780d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.699+0000 7ff67c9b7640 1 -- 172.21.15.146:0/3215353201 shutdown_connections 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.699+0000 7ff67c9b7640 1 -- 172.21.15.146:0/3215353201 wait complete. 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.701+0000 7ff67c9b7640 1 Processor -- start 2024-04-25T15:19:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.701+0000 7ff67c9b7640 1 -- start start 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67c9b7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6780b6610 0x7ff678149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67c9b7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff67815ffa0 0x7ff67815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67c9b7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 0x7ff678160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67c9b7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6780d1da0 con 0x7ff6781f5e80 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67c9b7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6781603b0 con 0x7ff67815ffa0 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67c9b7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff6780d1f20 con 0x7ff6780b6610 2024-04-25T15:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 0x7ff678160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 0x7ff678160c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36792/0 (socket says 172.21.15.146:36792) 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 -- 172.21.15.146:0/264568234 learned_addr learned my addr 172.21.15.146:0/264568234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67659c640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6780b6610 0x7ff678149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff675d9b640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff67815ffa0 0x7ff67815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 -- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6780b6610 msgr2=0x7ff678149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6780b6610 0x7ff678149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 -- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff67815ffa0 msgr2=0x7ff67815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff67815ffa0 0x7ff67815abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 -- 172.21.15.146:0/264568234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff678162bc0 con 0x7ff6781f5e80 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff67659c640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6780b6610 0x7ff678149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.702+0000 7ff676d9d640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 0x7ff678160c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff66800e7c0 tx=0x7ff66800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.703+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668017070 con 0x7ff6781f5e80 2024-04-25T15:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.703+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66800c730 con 0x7ff6781f5e80 2024-04-25T15:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.703+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6781623d0 con 0x7ff6781f5e80 2024-04-25T15:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.703+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66800cb20 con 0x7ff6781f5e80 2024-04-25T15:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.703+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff678161e10 con 0x7ff6781f5e80 2024-04-25T15:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.704+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff678162930 con 0x7ff6781f5e80 2024-04-25T15:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.704+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff66801a070 con 0x7ff6781f5e80 2024-04-25T15:19:57.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.705+0000 7ff666ffd640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6580744e0 0x7ff6580769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:57.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.705+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff668095360 con 0x7ff6781f5e80 2024-04-25T15:19:57.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.705+0000 7ff67659c640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6580744e0 0x7ff6580769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:57.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.705+0000 7ff67659c640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6580744e0 0x7ff6580769f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff6700012e0 tx=0x7ff670009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:57.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.709+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff66805f7b0 con 0x7ff6781f5e80 2024-04-25T15:19:57.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.895+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff67806a0e0 con 0x7ff6781f5e80 2024-04-25T15:19:57.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.896+0000 7ff666ffd640 1 -- 172.21.15.146:0/264568234 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7ff66805f0b0 con 0x7ff6781f5e80 2024-04-25T15:19:57.897 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:19:57.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6580744e0 msgr2=0x7ff6580769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6580744e0 0x7ff6580769f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff6700012e0 tx=0x7ff670009040 comp rx=0 tx=0).stop 2024-04-25T15:19:57.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 msgr2=0x7ff678160c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:57.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 0x7ff678160c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff66800e7c0 tx=0x7ff66800ec80 comp rx=0 tx=0).stop 2024-04-25T15:19:57.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 shutdown_connections 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6580744e0 0x7ff6580769f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6781f5e80 0x7ff678160c10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff67815ffa0 0x7ff67815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 --2- 172.21.15.146:0/264568234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6780b6610 0x7ff678149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 >> 172.21.15.146:0/264568234 conn(0x7ff6780d0240 msgr2=0x7ff6780b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 shutdown_connections 2024-04-25T15:19:57.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:57.898+0000 7ff67c9b7640 1 -- 172.21.15.146:0/264568234 wait complete. 2024-04-25T15:19:57.910 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:19:57.910 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:19:57.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:19:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.072+0000 7f4c6a242640 1 Processor -- start 2024-04-25T15:19:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.073+0000 7f4c6a242640 1 -- start start 2024-04-25T15:19:58.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c6a242640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c640cc280 0x7f4c64070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c6a242640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c640cd490 con 0x7f4c641618f0 2024-04-25T15:19:58.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c6a242640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c640711d0 con 0x7f4c6406c8c0 2024-04-25T15:19:58.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c6a242640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4c6415bcd0 con 0x7f4c640cc280 2024-04-25T15:19:58.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c62ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4c641618f0 0x7f4c64161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54202/0 (socket says 172.21.15.146:54202) 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c62ffd640 1 -- 172.21.15.146:0/1632750243 learned_addr learned my addr 172.21.15.146:0/1632750243 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c637fe640 1 --2- 172.21.15.146:0/1632750243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c640cc280 0x7f4c64070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611508053 0 0) 0x7f4c640cd490 con 0x7f4c641618f0 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.074+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c003520 con 0x7f4c641618f0 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890347998 0 0) 0x7f4c640711d0 con 0x7f4c6406c8c0 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c640cd490 con 0x7f4c6406c8c0 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3828035561 0 0) 0x7f4c4c003520 con 0x7f4c641618f0 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c637fe640 1 -- 172.21.15.146:0/1632750243 >> v1:172.21.15.177:6789/0 conn(0x7f4c6406c8c0 legacy=0x7f4c6406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c637fe640 1 -- 172.21.15.146:0/1632750243 >> v1:172.21.15.146:6789/0 conn(0x7f4c641618f0 legacy=0x7f4c64161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c637fe640 1 -- 172.21.15.146:0/1632750243 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c640cc880 con 0x7f4c640cc280 2024-04-25T15:19:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.075+0000 7f4c637fe640 1 --2- 172.21.15.146:0/1632750243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c640cc280 0x7f4c64070c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c540097d0 tx=0x7f4c5402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae921bb23a6ad7b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54037070 con 0x7f4c640cc280 2024-04-25T15:19:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c5402ab10 con 0x7f4c640cc280 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c627fc640 1 -- 172.21.15.146:0/1632750243 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c58001670 con 0x7f4c640cc280 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 -- 172.21.15.146:0/1632750243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c640cc280 msgr2=0x7f4c64070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 --2- 172.21.15.146:0/1632750243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c640cc280 0x7f4c64070c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c540097d0 tx=0x7f4c5402a080 comp rx=0 tx=0).stop 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 -- 172.21.15.146:0/1632750243 shutdown_connections 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 --2- 172.21.15.146:0/1632750243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c640cc280 0x7f4c64070c90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 -- 172.21.15.146:0/1632750243 >> 172.21.15.146:0/1632750243 conn(0x7f4c640ce0c0 msgr2=0x7f4c640ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 -- 172.21.15.146:0/1632750243 shutdown_connections 2024-04-25T15:19:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.076+0000 7f4c6a242640 1 -- 172.21.15.146:0/1632750243 wait complete. 2024-04-25T15:19:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.078+0000 7f4c6a242640 1 Processor -- start 2024-04-25T15:19:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.078+0000 7f4c6a242640 1 -- start start 2024-04-25T15:19:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c6406c8c0 0x7f4c641502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c640cc280 0x7f4c640b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 0x7f4c640b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c640cd120 con 0x7f4c641618f0 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c640cce40 con 0x7f4c640cc280 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4c640b0940 con 0x7f4c6406c8c0 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 0x7f4c640b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 0x7f4c640b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36794/0 (socket says 172.21.15.146:36794) 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 -- 172.21.15.146:0/341917697 learned_addr learned my addr 172.21.15.146:0/341917697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c637fe640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c6406c8c0 0x7f4c641502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c62ffd640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c640cc280 0x7f4c640b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 -- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c6406c8c0 msgr2=0x7f4c641502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c6406c8c0 0x7f4c641502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 -- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c640cc280 msgr2=0x7f4c640b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c640cc280 0x7f4c640b59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 -- 172.21.15.146:0/341917697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c640b1170 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c637fe640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c6406c8c0 0x7f4c641502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c62ffd640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c640cc280 0x7f4c640b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c63fff640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 0x7f4c640b2740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00c6b0 tx=0x7f4c5c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c003b70 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c5c003d10 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c640b6660 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c005310 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.080+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c640b1410 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.081+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c6406a0e0 con 0x7f4c641618f0 2024-04-25T15:19:58.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.082+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4c5c0054b0 con 0x7f4c641618f0 2024-04-25T15:19:58.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.082+0000 7f4c4bfff640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4c500744b0 0x7f4c500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.083+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4c5c095350 con 0x7f4c641618f0 2024-04-25T15:19:58.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.083+0000 7f4c637fe640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4c500744b0 0x7f4c500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.083+0000 7f4c637fe640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4c500744b0 0x7f4c500769c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c54034000 tx=0x7f4c5402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.087+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c5c05f8c0 con 0x7f4c641618f0 2024-04-25T15:19:58.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.261+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c6409f720 con 0x7f4c641618f0 2024-04-25T15:19:58.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.262+0000 7f4c4bfff640 1 -- 172.21.15.146:0/341917697 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7f4c5c05f1c0 con 0x7f4c641618f0 2024-04-25T15:19:58.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:19:58.263 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:19:58.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4c500744b0 msgr2=0x7f4c500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4c500744b0 0x7f4c500769c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c54034000 tx=0x7f4c5402e000 comp rx=0 tx=0).stop 2024-04-25T15:19:58.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 msgr2=0x7f4c640b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 0x7f4c640b2740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00c6b0 tx=0x7f4c5c00cb70 comp rx=0 tx=0).stop 2024-04-25T15:19:58.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 shutdown_connections 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4c500744b0 0x7f4c500769c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c641618f0 0x7f4c640b2740 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c640cc280 0x7f4c640b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 --2- 172.21.15.146:0/341917697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c6406c8c0 0x7f4c641502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 >> 172.21.15.146:0/341917697 conn(0x7f4c640ce0c0 msgr2=0x7f4c640afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 shutdown_connections 2024-04-25T15:19:58.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.264+0000 7f4c6a242640 1 -- 172.21.15.146:0/341917697 wait complete. 2024-04-25T15:19:58.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:19:58.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:19:58.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.387+0000 7f269faa7640 1 Processor -- start 2024-04-25T15:19:58.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.387+0000 7f269faa7640 1 -- start start 2024-04-25T15:19:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269faa7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f269806dc70 0x7f2698161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269faa7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2698162180 0x7f2698157870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269faa7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269815e450 con 0x7f2698070b20 2024-04-25T15:19:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269faa7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f269815ed80 con 0x7f2698162180 2024-04-25T15:19:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269faa7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f269815e1f0 con 0x7f269806dc70 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269e044640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2698162180 0x7f2698157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269d843640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2698070b20 0x7f269806d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54214/0 (socket says 172.21.15.146:54214) 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269e044640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2698162180 0x7f2698157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36806/0 (socket says 172.21.15.146:36806) 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269d843640 1 -- 172.21.15.146:0/1676381517 learned_addr learned my addr 172.21.15.146:0/1676381517 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269d042640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f269806dc70 0x7f2698161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269e044640 1 -- 172.21.15.146:0/1676381517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f269806dc70 msgr2=0x7f2698161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269e044640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f269806dc70 0x7f2698161c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269e044640 1 -- 172.21.15.146:0/1676381517 >> v1:172.21.15.146:6789/0 conn(0x7f2698070b20 legacy=0x7f269806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269e044640 1 -- 172.21.15.146:0/1676381517 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269815e730 con 0x7f2698162180 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269c841640 1 -- 172.21.15.146:0/1676381517 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 545073561 0 0) 0x7f269815e450 con 0x7f2698070b20 2024-04-25T15:19:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.389+0000 7f269d042640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f269806dc70 0x7f2698161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.390+0000 7f269e044640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2698162180 0x7f2698157870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26900097d0 tx=0x7f269002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfce82f3beb7da88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.390+0000 7f269c841640 1 -- 172.21.15.146:0/1676381517 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690037070 con 0x7f2698162180 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.390+0000 7f269c841640 1 -- 172.21.15.146:0/1676381517 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f269002ae40 con 0x7f2698162180 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.390+0000 7f269c841640 1 -- 172.21.15.146:0/1676381517 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690032660 con 0x7f2698162180 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 -- 172.21.15.146:0/1676381517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2698162180 msgr2=0x7f2698157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2698162180 0x7f2698157870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f26900097d0 tx=0x7f269002a130 comp rx=0 tx=0).stop 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 -- 172.21.15.146:0/1676381517 shutdown_connections 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2698162180 0x7f2698157870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 --2- 172.21.15.146:0/1676381517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f269806dc70 0x7f2698161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 -- 172.21.15.146:0/1676381517 >> 172.21.15.146:0/1676381517 conn(0x7f26980d1c10 msgr2=0x7f26980733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 -- 172.21.15.146:0/1676381517 shutdown_connections 2024-04-25T15:19:58.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.391+0000 7f269faa7640 1 -- 172.21.15.146:0/1676381517 wait complete. 2024-04-25T15:19:58.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.393+0000 7f269faa7640 1 Processor -- start 2024-04-25T15:19:58.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.393+0000 7f269faa7640 1 -- start start 2024-04-25T15:19:58.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269faa7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2698070b20 0x7f269814ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269faa7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2698162180 0x7f269815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269faa7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 0x7f269806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269faa7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2698070f30 con 0x7f269814eb50 2024-04-25T15:19:58.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269faa7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2698162590 con 0x7f2698162180 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269faa7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f26980710b0 con 0x7f2698070b20 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 0x7f269806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 0x7f269806caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36812/0 (socket says 172.21.15.146:36812) 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 -- 172.21.15.146:0/16680806 learned_addr learned my addr 172.21.15.146:0/16680806 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269d843640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2698070b20 0x7f269814ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269d042640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2698162180 0x7f269815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 -- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2698070b20 msgr2=0x7f269814ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2698070b20 0x7f269814ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 -- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2698162180 msgr2=0x7f269815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2698162180 0x7f269815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.394+0000 7f269e044640 1 -- 172.21.15.146:0/16680806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269814b3d0 con 0x7f269814eb50 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.395+0000 7f269e044640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 0x7f269806caf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f269002a720 tx=0x7f26900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.395+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690041030 con 0x7f269814eb50 2024-04-25T15:19:58.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.395+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f269002ac40 con 0x7f269814eb50 2024-04-25T15:19:58.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.395+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269814b6b0 con 0x7f269814eb50 2024-04-25T15:19:58.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.395+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690037040 con 0x7f269814eb50 2024-04-25T15:19:58.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.395+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2698148ab0 con 0x7f269814eb50 2024-04-25T15:19:58.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.396+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269814b140 con 0x7f269814eb50 2024-04-25T15:19:58.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.396+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2690044070 con 0x7f269814eb50 2024-04-25T15:19:58.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.397+0000 7f268e7fc640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f267c0744b0 0x7f267c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.397+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f26900b5a20 con 0x7f269814eb50 2024-04-25T15:19:58.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.397+0000 7f269d843640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f267c0744b0 0x7f267c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.397+0000 7f269d843640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f267c0744b0 0x7f267c0769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f26940012e0 tx=0x7f2694001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.401+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f269007f820 con 0x7f269814eb50 2024-04-25T15:19:58.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.579+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f269806a0e0 con 0x7f269814eb50 2024-04-25T15:19:58.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.580+0000 7f268e7fc640 1 -- 172.21.15.146:0/16680806 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f2690083b20 con 0x7f269814eb50 2024-04-25T15:19:58.581 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f267c0744b0 msgr2=0x7f267c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f267c0744b0 0x7f267c0769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f26940012e0 tx=0x7f2694001630 comp rx=0 tx=0).stop 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 msgr2=0x7f269806caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 0x7f269806caf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f269002a720 tx=0x7f26900046c0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 shutdown_connections 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f267c0744b0 0x7f267c0769c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f269814eb50 0x7f269806caf0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2698162180 0x7f269815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 --2- 172.21.15.146:0/16680806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2698070b20 0x7f269814ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 >> 172.21.15.146:0/16680806 conn(0x7f26980d1c10 msgr2=0x7f269815f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:58.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 shutdown_connections 2024-04-25T15:19:58.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.582+0000 7f269faa7640 1 -- 172.21.15.146:0/16680806 wait complete. 2024-04-25T15:19:58.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:19:58.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.703+0000 7ff1fac69640 1 Processor -- start 2024-04-25T15:19:58.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.703+0000 7ff1fac69640 1 -- start start 2024-04-25T15:19:58.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.704+0000 7ff1fac69640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1f40b4c30 0x7ff1f40b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.704+0000 7ff1fac69640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff1f40b9110 0x7ff1f41600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.704+0000 7ff1fac69640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f40b9520 con 0x7ff1f40d0a30 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.704+0000 7ff1fac69640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f406cc00 con 0x7ff1f40b4c30 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.704+0000 7ff1fac69640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f406c5f0 con 0x7ff1f40b9110 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1f40b4c30 0x7ff1f40b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1f40b4c30 0x7ff1f40b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36814/0 (socket says 172.21.15.146:36814) 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 -- 172.21.15.146:0/603269994 learned_addr learned my addr 172.21.15.146:0/603269994 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f9206640 1 --1- 172.21.15.146:0/603269994 >> v1:172.21.15.146:6789/0 conn(0x7ff1f40d0a30 0x7ff1f40d0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54220/0 (socket says 172.21.15.146:54220) 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f3fff640 1 --2- 172.21.15.146:0/603269994 >> v2:172.21.15.177:3300/0 conn(0x7ff1f40b9110 0x7ff1f41600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 -- 172.21.15.146:0/603269994 >> v2:172.21.15.177:3300/0 conn(0x7ff1f40b9110 msgr2=0x7ff1f41600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 --2- 172.21.15.146:0/603269994 >> v2:172.21.15.177:3300/0 conn(0x7ff1f40b9110 0x7ff1f41600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 -- 172.21.15.146:0/603269994 >> v1:172.21.15.146:6789/0 conn(0x7ff1f40d0a30 legacy=0x7ff1f40d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f8a05640 1 -- 172.21.15.146:0/603269994 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f41647b0 con 0x7ff1f40b4c30 2024-04-25T15:19:58.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f37fe640 1 -- 172.21.15.146:0/603269994 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293031749 0 0) 0x7ff1f40b9520 con 0x7ff1f40d0a30 2024-04-25T15:19:58.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.705+0000 7ff1f3fff640 1 --2- 172.21.15.146:0/603269994 >> v2:172.21.15.177:3300/0 conn(0x7ff1f40b9110 0x7ff1f41600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:19:58.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.706+0000 7ff1f8a05640 1 --2- 172.21.15.146:0/603269994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40b4c30 0x7ff1f40b4670 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40b4be0 tx=0x7ff1e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42afae472cf80086 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.706+0000 7ff1f37fe640 1 -- 172.21.15.146:0/603269994 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4037070 con 0x7ff1f40b4c30 2024-04-25T15:19:58.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.706+0000 7ff1f37fe640 1 -- 172.21.15.146:0/603269994 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e4032440 con 0x7ff1f40b4c30 2024-04-25T15:19:58.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.706+0000 7ff1f37fe640 1 -- 172.21.15.146:0/603269994 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4032810 con 0x7ff1f40b4c30 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 -- 172.21.15.146:0/603269994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40b4c30 msgr2=0x7ff1f40b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 --2- 172.21.15.146:0/603269994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40b4c30 0x7ff1f40b4670 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40b4be0 tx=0x7ff1e402a270 comp rx=0 tx=0).stop 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 -- 172.21.15.146:0/603269994 shutdown_connections 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 --2- 172.21.15.146:0/603269994 >> v2:172.21.15.177:3300/0 conn(0x7ff1f40b9110 0x7ff1f41600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 --2- 172.21.15.146:0/603269994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40b4c30 0x7ff1f40b4670 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 -- 172.21.15.146:0/603269994 >> 172.21.15.146:0/603269994 conn(0x7ff1f40b7970 msgr2=0x7ff1f40b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 -- 172.21.15.146:0/603269994 shutdown_connections 2024-04-25T15:19:58.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.707+0000 7ff1fac69640 1 -- 172.21.15.146:0/603269994 wait complete. 2024-04-25T15:19:58.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.708+0000 7ff1fac69640 1 Processor -- start 2024-04-25T15:19:58.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.708+0000 7ff1fac69640 1 -- start start 2024-04-25T15:19:58.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1fac69640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1f40b4c30 0x7ff1f414cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1fac69640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1f40b9110 0x7ff1f40ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1fac69640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 0x7ff1f40cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1fac69640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f40d16b0 con 0x7ff1f40d0a30 2024-04-25T15:19:58.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1fac69640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f4164a90 con 0x7ff1f40b4c30 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1fac69640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f406d090 con 0x7ff1f40b9110 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 0x7ff1f40cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 0x7ff1f40cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36818/0 (socket says 172.21.15.146:36818) 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 -- 172.21.15.146:0/2977524446 learned_addr learned my addr 172.21.15.146:0/2977524446 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f3fff640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1f40b9110 0x7ff1f40ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f8a05640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1f40b4c30 0x7ff1f414cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 -- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1f40b9110 msgr2=0x7ff1f40ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1f40b9110 0x7ff1f40ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 -- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1f40b4c30 msgr2=0x7ff1f414cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1f40b4c30 0x7ff1f414cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.710+0000 7ff1f9206640 1 -- 172.21.15.146:0/2977524446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f41eaa10 con 0x7ff1f40d0a30 2024-04-25T15:19:58.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.711+0000 7ff1f9206640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 0x7ff1f40cc120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00c730 tx=0x7ff1ec00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.711+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec018070 con 0x7ff1f40d0a30 2024-04-25T15:19:58.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.711+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ec00ce40 con 0x7ff1f40d0a30 2024-04-25T15:19:58.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.711+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec00a860 con 0x7ff1f40d0a30 2024-04-25T15:19:58.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.711+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f4150700 con 0x7ff1f40d0a30 2024-04-25T15:19:58.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.711+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f41eaf90 con 0x7ff1f40d0a30 2024-04-25T15:19:58.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.712+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f406a0e0 con 0x7ff1f40d0a30 2024-04-25T15:19:58.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.712+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff1ec01b070 con 0x7ff1f40d0a30 2024-04-25T15:19:58.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.713+0000 7ff1f17fa640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1e00744e0 0x7ff1e00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:58.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.713+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff1ec094ac0 con 0x7ff1f40d0a30 2024-04-25T15:19:58.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.713+0000 7ff1f8a05640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1e00744e0 0x7ff1e00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:58.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.713+0000 7ff1f8a05640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1e00744e0 0x7ff1e00769f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4034000 tx=0x7ff1e402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.718+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1ec05ef10 con 0x7ff1f40d0a30 2024-04-25T15:19:58.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.893+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1f40d3fa0 con 0x7ff1f40d0a30 2024-04-25T15:19:58.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.894+0000 7ff1f17fa640 1 -- 172.21.15.146:0/2977524446 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ff1ec05e810 con 0x7ff1f40d0a30 2024-04-25T15:19:58.895 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:19:58.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1e00744e0 msgr2=0x7ff1e00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1e00744e0 0x7ff1e00769f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4034000 tx=0x7ff1e402e000 comp rx=0 tx=0).stop 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 msgr2=0x7ff1f40cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 0x7ff1f40cc120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00c730 tx=0x7ff1ec00cbf0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 shutdown_connections 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1e00744e0 0x7ff1e00769f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1f40d0a30 0x7ff1f40cc120 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1f40b9110 0x7ff1f40ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 --2- 172.21.15.146:0/2977524446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1f40b4c30 0x7ff1f414cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 >> 172.21.15.146:0/2977524446 conn(0x7ff1f40b7970 msgr2=0x7ff1f41631b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 shutdown_connections 2024-04-25T15:19:58.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:58.896+0000 7ff1fac69640 1 -- 172.21.15.146:0/2977524446 wait complete. 2024-04-25T15:19:58.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:19:59.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.019+0000 7f7534677640 1 Processor -- start 2024-04-25T15:19:59.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.019+0000 7f7534677640 1 -- start start 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7534677640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f752c09bd80 0x7f752c0ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7534677640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f752c0ce920 con 0x7f752c15e070 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7534677640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f752c0cec00 con 0x7f752c0d2680 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7534677640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f752c0ceee0 con 0x7f752c09bd80 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7532413640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f752c15e070 0x7f752c0cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54228/0 (socket says 172.21.15.146:54228) 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7532413640 1 -- 172.21.15.146:0/2772916766 learned_addr learned my addr 172.21.15.146:0/2772916766 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:19:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531c12640 1 --2- 172.21.15.146:0/2772916766 >> v2:172.21.15.177:3300/0 conn(0x7f752c09bd80 0x7f752c0ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449312519 0 0) 0x7f752c0ce920 con 0x7f752c15e070 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7518003520 con 0x7f752c15e070 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 504969704 0 0) 0x7f7518003520 con 0x7f752c15e070 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 >> v2:172.21.15.177:3300/0 conn(0x7f752c09bd80 msgr2=0x7f752c0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 --2- 172.21.15.146:0/2772916766 >> v2:172.21.15.177:3300/0 conn(0x7f752c09bd80 0x7f752c0ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 >> v1:172.21.15.177:6789/0 conn(0x7f752c0d2680 legacy=0x7f752c0d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c0cf1c0 con 0x7f752c15e070 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f75280017f0 con 0x7f752c15e070 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 >> v1:172.21.15.146:6789/0 conn(0x7f752c15e070 legacy=0x7f752c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180039b0 0x7f7518003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7518004800 0x7f7518004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75180051c0 0x7f751800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f752c0ceee0 con 0x7f75180051c0 2024-04-25T15:19:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f751800c160 con 0x7f7518004800 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f751800c360 con 0x7f75180039b0 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7532c14640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75180051c0 0x7f751800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7532413640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180039b0 0x7f7518003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.021+0000 7f7531c12640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7518004800 0x7f7518004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 -- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180039b0 msgr2=0x7f7518003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180039b0 0x7f7518003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 -- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7518004800 msgr2=0x7f7518004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7518004800 0x7f7518004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 -- 172.21.15.146:0/2772916766 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7518003520 con 0x7f75180051c0 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 -- 172.21.15.146:0/2772916766 reap_dead start 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.022+0000 7f7532c14640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75180051c0 0x7f751800b9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7524001e20 tx=0x7f7518005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398e4d5574b5d98d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524001a70 con 0x7f75180051c0 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f752402ad70 con 0x7f75180051c0 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7531411640 1 -- 172.21.15.146:0/2772916766 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524003bb0 con 0x7f75180051c0 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 -- 172.21.15.146:0/2772916766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75180051c0 msgr2=0x7f751800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75180051c0 0x7f751800b9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7524001e20 tx=0x7f7518005830 comp rx=0 tx=0).stop 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 -- 172.21.15.146:0/2772916766 shutdown_connections 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 --2- 172.21.15.146:0/2772916766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75180051c0 0x7f751800b9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 -- 172.21.15.146:0/2772916766 >> 172.21.15.146:0/2772916766 conn(0x7f752c0b7400 msgr2=0x7f752c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:59.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 -- 172.21.15.146:0/2772916766 shutdown_connections 2024-04-25T15:19:59.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.023+0000 7f7534677640 1 -- 172.21.15.146:0/2772916766 wait complete. 2024-04-25T15:19:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.025+0000 7f7534677640 1 Processor -- start 2024-04-25T15:19:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.025+0000 7f7534677640 1 -- start start 2024-04-25T15:19:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7534677640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180051c0 0x7f752c15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7534677640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 0x7f752c156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7534677640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f752c156a80 0x7f752c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7534677640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f752c0d2c10 con 0x7f752c154e40 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7534677640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f752c0d2a90 con 0x7f752c156a80 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7534677640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f752c0d2d90 con 0x7f75180051c0 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7531c12640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 0x7f752c156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7531c12640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 0x7f752c156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36834/0 (socket says 172.21.15.146:36834) 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7531c12640 1 -- 172.21.15.146:0/3993906476 learned_addr learned my addr 172.21.15.146:0/3993906476 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7532413640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180051c0 0x7f752c15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.026+0000 7f7532c14640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f752c156a80 0x7f752c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7531c12640 1 -- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180051c0 msgr2=0x7f752c15d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7531c12640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180051c0 0x7f752c15d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7531c12640 1 -- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f752c156a80 msgr2=0x7f752c0b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7531c12640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f752c156a80 0x7f752c0b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7531c12640 1 -- 172.21.15.146:0/3993906476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c156e90 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7531c12640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 0x7f752c156540 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f751c0118e0 tx=0x7f751c011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c00f9d0 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751c00fb70 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c014440 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c0ca230 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.027+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c0ca7f0 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.028+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752c06a0e0 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.029+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f751c012040 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.029+0000 7f7522ffd640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75100744e0 0x7f75100769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.029+0000 7f7532413640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75100744e0 0x7f75100769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.029+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f751c095910 con 0x7f752c154e40 2024-04-25T15:19:59.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.030+0000 7f7532413640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75100744e0 0x7f75100769f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7528000ef0 tx=0x7f7528009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.034+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f752c156e90 con 0x7f752c154e40 2024-04-25T15:19:59.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.210+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f752c0cd030 con 0x7f752c154e40 2024-04-25T15:19:59.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.211+0000 7f7522ffd640 1 -- 172.21.15.146:0/3993906476 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f751c019030 con 0x7f752c154e40 2024-04-25T15:19:59.212 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75100744e0 msgr2=0x7f75100769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75100744e0 0x7f75100769f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7528000ef0 tx=0x7f7528009040 comp rx=0 tx=0).stop 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 msgr2=0x7f752c156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 0x7f752c156540 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f751c0118e0 tx=0x7f751c011ce0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 shutdown_connections 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f75100744e0 0x7f75100769f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f752c156a80 0x7f752c0b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f752c154e40 0x7f752c156540 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 --2- 172.21.15.146:0/3993906476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75180051c0 0x7f752c15d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 >> 172.21.15.146:0/3993906476 conn(0x7f752c0b7400 msgr2=0x7f752c161980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 shutdown_connections 2024-04-25T15:19:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.213+0000 7f7534677640 1 -- 172.21.15.146:0/3993906476 wait complete. 2024-04-25T15:19:59.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:19:59.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.337+0000 7f43c639f640 1 Processor -- start 2024-04-25T15:19:59.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.337+0000 7f43c639f640 1 -- start start 2024-04-25T15:19:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c639f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f43c00cf340 0x7f43c00cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c639f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f43c00b7350 0x7f43c00b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c639f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c006def0 0x7f43c006e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c639f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f43c00b7ca0 con 0x7f43c00b7350 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c639f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f43c00d2d10 con 0x7f43c00cf340 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c639f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f43c00d2ff0 con 0x7f43c006def0 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c4b9c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f43c00b7350 0x7f43c00b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c4b9c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f43c00b7350 0x7f43c00b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36844/0 (socket says 172.21.15.146:36844) 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c4b9c640 1 -- 172.21.15.146:0/1721935301 learned_addr learned my addr 172.21.15.146:0/1721935301 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c5b9e640 1 --2- 172.21.15.146:0/1721935301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c006def0 0x7f43c006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.338+0000 7f43c539d640 1 --2- 172.21.15.146:0/1721935301 >> v2:172.21.15.177:3300/0 conn(0x7f43c00cf340 0x7f43c00cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.339+0000 7f43c4b9c640 1 -- 172.21.15.146:0/1721935301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c006def0 msgr2=0x7f43c006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.339+0000 7f43c4b9c640 1 --2- 172.21.15.146:0/1721935301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c006def0 0x7f43c006e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.339+0000 7f43c4b9c640 1 -- 172.21.15.146:0/1721935301 >> v2:172.21.15.177:3300/0 conn(0x7f43c00cf340 msgr2=0x7f43c00cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.339+0000 7f43c4b9c640 1 --2- 172.21.15.146:0/1721935301 >> v2:172.21.15.177:3300/0 conn(0x7f43c00cf340 0x7f43c00cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.339+0000 7f43c4b9c640 1 -- 172.21.15.146:0/1721935301 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c00d32d0 con 0x7f43c00b7350 2024-04-25T15:19:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.339+0000 7f43c4b9c640 1 --2- 172.21.15.146:0/1721935301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00b7350 0x7f43c00b7760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43c00d3630 tx=0x7f43b002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1f6c44537ee8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43b7fff640 1 -- 172.21.15.146:0/1721935301 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0037070 con 0x7f43c00b7350 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43b7fff640 1 -- 172.21.15.146:0/1721935301 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b0032440 con 0x7f43c00b7350 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43b7fff640 1 -- 172.21.15.146:0/1721935301 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0032810 con 0x7f43c00b7350 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 -- 172.21.15.146:0/1721935301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00b7350 msgr2=0x7f43c00b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 --2- 172.21.15.146:0/1721935301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00b7350 0x7f43c00b7760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f43c00d3630 tx=0x7f43b002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 -- 172.21.15.146:0/1721935301 shutdown_connections 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 --2- 172.21.15.146:0/1721935301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c006def0 0x7f43c006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 --2- 172.21.15.146:0/1721935301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00b7350 0x7f43c00b7760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 --2- 172.21.15.146:0/1721935301 >> v2:172.21.15.177:3300/0 conn(0x7f43c00cf340 0x7f43c00cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 -- 172.21.15.146:0/1721935301 >> 172.21.15.146:0/1721935301 conn(0x7f43c009b2b0 msgr2=0x7f43c009b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 -- 172.21.15.146:0/1721935301 shutdown_connections 2024-04-25T15:19:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.340+0000 7f43c639f640 1 -- 172.21.15.146:0/1721935301 wait complete. 2024-04-25T15:19:59.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.342+0000 7f43c639f640 1 Processor -- start 2024-04-25T15:19:59.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.342+0000 7f43c639f640 1 -- start start 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.343+0000 7f43c639f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c006def0 0x7f43c01f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.343+0000 7f43c639f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c00b7350 0x7f43c015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c639f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 0x7f43c006cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c639f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f43c00cfe60 con 0x7f43c00cf340 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c639f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43c00b24d0 con 0x7f43c006def0 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c639f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f43c00d1d40 con 0x7f43c00b7350 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 0x7f43c006cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 0x7f43c006cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36860/0 (socket says 172.21.15.146:36860) 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 -- 172.21.15.146:0/2909954808 learned_addr learned my addr 172.21.15.146:0/2909954808 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c4b9c640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c00b7350 0x7f43c015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c539d640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c006def0 0x7f43c01f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 -- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c00b7350 msgr2=0x7f43c015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c00b7350 0x7f43c015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 -- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c006def0 msgr2=0x7f43c01f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c006def0 0x7f43c01f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 -- 172.21.15.146:0/2909954808 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c015ccd0 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c5b9e640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 0x7f43c006cfc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f43b800e7c0 tx=0x7f43b800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8017070 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b800c730 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c0158590 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.344+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b800cb40 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.345+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c0158b50 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.345+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c0050740 con 0x7f43c00cf340 2024-04-25T15:19:59.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.346+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f43b801a070 con 0x7f43c00cf340 2024-04-25T15:19:59.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.347+0000 7f43b5ffb640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43a40744e0 0x7f43a40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.347+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f43b80955b0 con 0x7f43c00cf340 2024-04-25T15:19:59.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.347+0000 7f43c539d640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43a40744e0 0x7f43a40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.347+0000 7f43c539d640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43a40744e0 0x7f43a40769f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f43bc011340 tx=0x7f43bc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.351+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43b805f440 con 0x7f43c00cf340 2024-04-25T15:19:59.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.526+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f43c015ca60 con 0x7f43c00cf340 2024-04-25T15:19:59.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.526+0000 7f43b5ffb640 1 -- 172.21.15.146:0/2909954808 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f43b8063740 con 0x7f43c00cf340 2024-04-25T15:19:59.527 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43a40744e0 msgr2=0x7f43a40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43a40744e0 0x7f43a40769f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f43bc011340 tx=0x7f43bc00b040 comp rx=0 tx=0).stop 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 msgr2=0x7f43c006cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 0x7f43c006cfc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f43b800e7c0 tx=0x7f43b800ec80 comp rx=0 tx=0).stop 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 shutdown_connections 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43a40744e0 0x7f43a40769f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c00cf340 0x7f43c006cfc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c00b7350 0x7f43c015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 --2- 172.21.15.146:0/2909954808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c006def0 0x7f43c01f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 >> 172.21.15.146:0/2909954808 conn(0x7f43c009b2b0 msgr2=0x7f43c015d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:59.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 shutdown_connections 2024-04-25T15:19:59.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.528+0000 7f43c639f640 1 -- 172.21.15.146:0/2909954808 wait complete. 2024-04-25T15:19:59.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:19:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.648+0000 7f601f93f640 1 Processor -- start 2024-04-25T15:19:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.649+0000 7f601f93f640 1 -- start start 2024-04-25T15:19:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601f93f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f601815ded0 0x7f60180b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601f93f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60180b3c10 0x7f60180b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601f93f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601809c2a0 con 0x7f60180d21f0 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601f93f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f60180d1d60 con 0x7f60180b3c10 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601f93f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f60180b4020 con 0x7f601815ded0 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60180b3c10 0x7f60180b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60180b3c10 0x7f60180b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36876/0 (socket says 172.21.15.146:36876) 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 -- 172.21.15.146:0/3734168132 learned_addr learned my addr 172.21.15.146:0/3734168132 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601ceda640 1 --2- 172.21.15.146:0/3734168132 >> v2:172.21.15.177:3300/0 conn(0x7f601815ded0 0x7f60180b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 -- 172.21.15.146:0/3734168132 >> v2:172.21.15.177:3300/0 conn(0x7f601815ded0 msgr2=0x7f60180b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 --2- 172.21.15.146:0/3734168132 >> v2:172.21.15.177:3300/0 conn(0x7f601815ded0 0x7f60180b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 -- 172.21.15.146:0/3734168132 >> v1:172.21.15.177:6789/0 conn(0x7f60180d21f0 legacy=0x7f601806f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:19:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.650+0000 7f601dedc640 1 -- 172.21.15.146:0/3734168132 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601809bce0 con 0x7f60180b3c10 2024-04-25T15:19:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.651+0000 7f601dedc640 1 --2- 172.21.15.146:0/3734168132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180b3c10 0x7f60180b21b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f601809ba30 tx=0x7f601002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7aa94150118ea5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.651+0000 7f600ffff640 1 -- 172.21.15.146:0/3734168132 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010037070 con 0x7f60180b3c10 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.651+0000 7f600ffff640 1 -- 172.21.15.146:0/3734168132 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6010032440 con 0x7f60180b3c10 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.651+0000 7f600ffff640 1 -- 172.21.15.146:0/3734168132 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010032810 con 0x7f60180b3c10 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 -- 172.21.15.146:0/3734168132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180b3c10 msgr2=0x7f60180b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 --2- 172.21.15.146:0/3734168132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180b3c10 0x7f60180b21b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f601809ba30 tx=0x7f601002a2c0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 -- 172.21.15.146:0/3734168132 shutdown_connections 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 --2- 172.21.15.146:0/3734168132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180b3c10 0x7f60180b21b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 --2- 172.21.15.146:0/3734168132 >> v2:172.21.15.177:3300/0 conn(0x7f601815ded0 0x7f60180b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 -- 172.21.15.146:0/3734168132 >> 172.21.15.146:0/3734168132 conn(0x7f60180b7530 msgr2=0x7f60180b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 -- 172.21.15.146:0/3734168132 shutdown_connections 2024-04-25T15:19:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.652+0000 7f601f93f640 1 -- 172.21.15.146:0/3734168132 wait complete. 2024-04-25T15:19:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.654+0000 7f601f93f640 1 Processor -- start 2024-04-25T15:19:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.654+0000 7f601f93f640 1 -- start start 2024-04-25T15:19:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601f93f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60180b3c10 0x7f6018153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601f93f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 0x7f601806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601f93f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f601815ded0 0x7f60180712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601f93f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f601815e2e0 con 0x7f60180d21f0 2024-04-25T15:19:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601f93f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60180d19f0 con 0x7f601815ded0 2024-04-25T15:19:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601f93f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f601809b400 con 0x7f60180b3c10 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601ceda640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 0x7f601806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601ceda640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 0x7f601806e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36884/0 (socket says 172.21.15.146:36884) 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601ceda640 1 -- 172.21.15.146:0/1842027389 learned_addr learned my addr 172.21.15.146:0/1842027389 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.655+0000 7f601d6db640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60180b3c10 0x7f6018153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601dedc640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f601815ded0 0x7f60180712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601ceda640 1 -- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60180b3c10 msgr2=0x7f6018153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601ceda640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60180b3c10 0x7f6018153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601ceda640 1 -- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f601815ded0 msgr2=0x7f60180712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601ceda640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f601815ded0 0x7f60180712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601ceda640 1 -- 172.21.15.146:0/1842027389 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601814d540 con 0x7f60180d21f0 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601dedc640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f601815ded0 0x7f60180712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601ceda640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 0x7f601806e840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f600800d5d0 tx=0x7f600800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008004240 con 0x7f60180d21f0 2024-04-25T15:19:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60080043e0 con 0x7f60180d21f0 2024-04-25T15:19:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008003830 con 0x7f60180d21f0 2024-04-25T15:19:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601814d820 con 0x7f60180d21f0 2024-04-25T15:19:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.656+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601814dde0 con 0x7f60180d21f0 2024-04-25T15:19:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.657+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601806a0e0 con 0x7f60180d21f0 2024-04-25T15:19:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.658+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f60080039d0 con 0x7f60180d21f0 2024-04-25T15:19:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.658+0000 7f600dffb640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ffc0744e0 0x7f5ffc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:19:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.658+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6008095790 con 0x7f60180d21f0 2024-04-25T15:19:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.658+0000 7f601d6db640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ffc0744e0 0x7f5ffc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:19:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.659+0000 7f601d6db640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ffc0744e0 0x7f5ffc0769f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f601400b450 tx=0x7f601400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:19:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.663+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f600805fbe0 con 0x7f60180d21f0 2024-04-25T15:19:59.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.848+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f601806ee00 con 0x7f60180d21f0 2024-04-25T15:19:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.849+0000 7f600dffb640 1 -- 172.21.15.146:0/1842027389 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f600805f4e0 con 0x7f60180d21f0 2024-04-25T15:19:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ffc0744e0 msgr2=0x7f5ffc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ffc0744e0 0x7f5ffc0769f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f601400b450 tx=0x7f601400d040 comp rx=0 tx=0).stop 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 msgr2=0x7f601806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 0x7f601806e840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f600800d5d0 tx=0x7f600800da90 comp rx=0 tx=0).stop 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 shutdown_connections 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ffc0744e0 0x7f5ffc0769f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f601815ded0 0x7f60180712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180d21f0 0x7f601806e840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 --2- 172.21.15.146:0/1842027389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60180b3c10 0x7f6018153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 >> 172.21.15.146:0/1842027389 conn(0x7f60180b7530 msgr2=0x7f60180cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 shutdown_connections 2024-04-25T15:19:59.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:19:59.851+0000 7f601f93f640 1 -- 172.21.15.146:0/1842027389 wait complete. 2024-04-25T15:19:59.863 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:19:59.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:19:59.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T15:19:59.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi146.stderr:ceph-fuse[41860]: fuse finished with error 0 and tester_r 0 2024-04-25T15:19:59.886 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:20:05.988 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:20:05.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:05.988 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T15:20:06.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:06.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.181+0000 7f0f2d764640 1 Processor -- start 2024-04-25T15:20:06.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.181+0000 7f0f2d764640 1 -- start start 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f2d764640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0f2806c550 0x7f0f2806daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f2d764640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0f280d4460 0x7f0f280b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f2d764640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2806ca50 con 0x7f0f280d0910 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f2d764640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0f2806d660 con 0x7f0f280d4460 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f2d764640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0f2806cd30 con 0x7f0f2806c550 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f267fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0f280d4460 0x7f0f280b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.182+0000 7f0f277fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0f280d0910 0x7f0f280d0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33948/0 (socket says 172.21.15.146:33948) 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f277fe640 1 -- 172.21.15.146:0/2741795789 learned_addr learned my addr 172.21.15.146:0/2741795789 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f267fc640 1 --2- 172.21.15.146:0/2741795789 >> v2:172.21.15.146:3300/0 conn(0x7f0f280d4460 0x7f0f280b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36692/0 (socket says 172.21.15.146:36692) 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f26ffd640 1 --2- 172.21.15.146:0/2741795789 >> v2:172.21.15.177:3300/0 conn(0x7f0f2806c550 0x7f0f2806daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f267fc640 1 -- 172.21.15.146:0/2741795789 >> v2:172.21.15.177:3300/0 conn(0x7f0f2806c550 msgr2=0x7f0f2806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:06.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f267fc640 1 --2- 172.21.15.146:0/2741795789 >> v2:172.21.15.177:3300/0 conn(0x7f0f2806c550 0x7f0f2806daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f267fc640 1 -- 172.21.15.146:0/2741795789 >> v1:172.21.15.146:6789/0 conn(0x7f0f280d0910 legacy=0x7f0f280d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f25ffb640 1 -- 172.21.15.146:0/2741795789 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 781645806 0 0) 0x7f0f2806ca50 con 0x7f0f280d0910 2024-04-25T15:20:06.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.183+0000 7f0f267fc640 1 -- 172.21.15.146:0/2741795789 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2806d010 con 0x7f0f280d4460 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f267fc640 1 --2- 172.21.15.146:0/2741795789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d4460 0x7f0f280b4e50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0f2808dfd0 tx=0x7f0f1c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da82b2f88b6055f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f25ffb640 1 -- 172.21.15.146:0/2741795789 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c037070 con 0x7f0f280d4460 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f25ffb640 1 -- 172.21.15.146:0/2741795789 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f1c032440 con 0x7f0f280d4460 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f25ffb640 1 -- 172.21.15.146:0/2741795789 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c032810 con 0x7f0f280d4460 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f2d764640 1 -- 172.21.15.146:0/2741795789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d4460 msgr2=0x7f0f280b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2741795789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d4460 0x7f0f280b4e50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0f2808dfd0 tx=0x7f0f1c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f2d764640 1 -- 172.21.15.146:0/2741795789 shutdown_connections 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2741795789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d4460 0x7f0f280b4e50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2741795789 >> v2:172.21.15.177:3300/0 conn(0x7f0f2806c550 0x7f0f2806daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.184+0000 7f0f2d764640 1 -- 172.21.15.146:0/2741795789 >> 172.21.15.146:0/2741795789 conn(0x7f0f280d59d0 msgr2=0x7f0f280d5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.185+0000 7f0f2d764640 1 -- 172.21.15.146:0/2741795789 shutdown_connections 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.185+0000 7f0f2d764640 1 -- 172.21.15.146:0/2741795789 wait complete. 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.186+0000 7f0f2d764640 1 Processor -- start 2024-04-25T15:20:06.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.186+0000 7f0f2d764640 1 -- start start 2024-04-25T15:20:06.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f2d764640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f2806c550 0x7f0f281eb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f2d764640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 0x7f0f28157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f2d764640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f280d4460 0x7f0f281f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f2d764640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0f280d15b0 con 0x7f0f280d0910 2024-04-25T15:20:06.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f2d764640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f280d1430 con 0x7f0f2806c550 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f2d764640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0f280b9020 con 0x7f0f280d4460 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 0x7f0f28157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 0x7f0f28157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36696/0 (socket says 172.21.15.146:36696) 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 -- 172.21.15.146:0/2638552589 learned_addr learned my addr 172.21.15.146:0/2638552589 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f26ffd640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f2806c550 0x7f0f281eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f277fe640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f280d4460 0x7f0f281f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 -- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f280d4460 msgr2=0x7f0f281f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f280d4460 0x7f0f281f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 -- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f2806c550 msgr2=0x7f0f281eb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f2806c550 0x7f0f281eb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 -- 172.21.15.146:0/2638552589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f281ec480 con 0x7f0f280d0910 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.188+0000 7f0f267fc640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 0x7f0f28157ad0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c000c00 tx=0x7f0f1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.189+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c037050 con 0x7f0f280d0910 2024-04-25T15:20:06.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.189+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f1c038040 con 0x7f0f280d0910 2024-04-25T15:20:06.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.189+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f281ec760 con 0x7f0f280d0910 2024-04-25T15:20:06.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.189+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c006e10 con 0x7f0f280d0910 2024-04-25T15:20:06.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.189+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f280b6fb0 con 0x7f0f280d0910 2024-04-25T15:20:06.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.190+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2806ca50 con 0x7f0f280d0910 2024-04-25T15:20:06.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.190+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0f1c030980 con 0x7f0f280d0910 2024-04-25T15:20:06.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.191+0000 7f0f0f7fe640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0f140744b0 0x7f0f140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.191+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0f1c0b5e60 con 0x7f0f280d0910 2024-04-25T15:20:06.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.191+0000 7f0f26ffd640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0f140744b0 0x7f0f140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.191+0000 7f0f26ffd640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0f140744b0 0x7f0f140769c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f280d0140 tx=0x7f0f18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.195+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f2806ca50 con 0x7f0f280d0910 2024-04-25T15:20:06.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.372+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f2806a0e0 con 0x7f0f280d0910 2024-04-25T15:20:06.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.373+0000 7f0f0f7fe640 1 -- 172.21.15.146:0/2638552589 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7f0f1c04c020 con 0x7f0f280d0910 2024-04-25T15:20:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:06.375 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:06.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0f140744b0 msgr2=0x7f0f140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0f140744b0 0x7f0f140769c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f280d0140 tx=0x7f0f18008040 comp rx=0 tx=0).stop 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 msgr2=0x7f0f28157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 0x7f0f28157ad0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c000c00 tx=0x7f0f1c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 shutdown_connections 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0f140744b0 0x7f0f140769c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f280d4460 0x7f0f281f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f280d0910 0x7f0f28157ad0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 --2- 172.21.15.146:0/2638552589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f2806c550 0x7f0f281eb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.375+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 >> 172.21.15.146:0/2638552589 conn(0x7f0f280d59d0 msgr2=0x7f0f2815ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.376+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 shutdown_connections 2024-04-25T15:20:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.376+0000 7f0f2d764640 1 -- 172.21.15.146:0/2638552589 wait complete. 2024-04-25T15:20:06.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:06.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.495+0000 7f153d601640 1 Processor -- start 2024-04-25T15:20:06.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.496+0000 7f153d601640 1 -- start start 2024-04-25T15:20:06.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f153d601640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f153815d030 0x7f153806f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f153d601640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f153806d560 0x7f153806d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f153d601640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153806deb0 con 0x7f153815a610 2024-04-25T15:20:06.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f153d601640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f153815dfd0 con 0x7f153806d560 2024-04-25T15:20:06.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f153d601640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f153815aa20 con 0x7f153815d030 2024-04-25T15:20:06.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f153806d560 0x7f153806d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f153806d560 0x7f153806d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36704/0 (socket says 172.21.15.146:36704) 2024-04-25T15:20:06.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 -- 172.21.15.146:0/1624007223 learned_addr learned my addr 172.21.15.146:0/1624007223 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:06.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15367fc640 1 --2- 172.21.15.146:0/1624007223 >> v2:172.21.15.177:3300/0 conn(0x7f153815d030 0x7f153806f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 -- 172.21.15.146:0/1624007223 >> v2:172.21.15.177:3300/0 conn(0x7f153815d030 msgr2=0x7f153806f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:06.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 --2- 172.21.15.146:0/1624007223 >> v2:172.21.15.177:3300/0 conn(0x7f153815d030 0x7f153806f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 -- 172.21.15.146:0/1624007223 >> v1:172.21.15.177:6789/0 conn(0x7f153815a610 legacy=0x7f15380ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.497+0000 7f15377fe640 1 -- 172.21.15.146:0/1624007223 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153815d730 con 0x7f153806d560 2024-04-25T15:20:06.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.498+0000 7f15377fe640 1 --2- 172.21.15.146:0/1624007223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153806d560 0x7f153806d970 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1538160560 tx=0x7f153002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42278636d91bf9b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f1535ffb640 1 -- 172.21.15.146:0/1624007223 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530036070 con 0x7f153806d560 2024-04-25T15:20:06.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f1535ffb640 1 -- 172.21.15.146:0/1624007223 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530031600 con 0x7f153806d560 2024-04-25T15:20:06.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f1535ffb640 1 -- 172.21.15.146:0/1624007223 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15300319d0 con 0x7f153806d560 2024-04-25T15:20:06.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f153d601640 1 -- 172.21.15.146:0/1624007223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153806d560 msgr2=0x7f153806d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f153d601640 1 --2- 172.21.15.146:0/1624007223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153806d560 0x7f153806d970 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1538160560 tx=0x7f153002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f153d601640 1 -- 172.21.15.146:0/1624007223 shutdown_connections 2024-04-25T15:20:06.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f153d601640 1 --2- 172.21.15.146:0/1624007223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153806d560 0x7f153806d970 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f153d601640 1 --2- 172.21.15.146:0/1624007223 >> v2:172.21.15.177:3300/0 conn(0x7f153815d030 0x7f153806f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.499+0000 7f153d601640 1 -- 172.21.15.146:0/1624007223 >> 172.21.15.146:0/1624007223 conn(0x7f15380731a0 msgr2=0x7f15380735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:06.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.500+0000 7f153d601640 1 -- 172.21.15.146:0/1624007223 shutdown_connections 2024-04-25T15:20:06.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.500+0000 7f153d601640 1 -- 172.21.15.146:0/1624007223 wait complete. 2024-04-25T15:20:06.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.501+0000 7f153d601640 1 Processor -- start 2024-04-25T15:20:06.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.501+0000 7f153d601640 1 -- start start 2024-04-25T15:20:06.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f153d601640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153806d560 0x7f15381410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f153d601640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153815a610 0x7f153813de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f153d601640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 0x7f153815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f153d601640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f15380b58c0 con 0x7f153815d030 2024-04-25T15:20:06.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f153d601640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f153815dcf0 con 0x7f153815a610 2024-04-25T15:20:06.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f153d601640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f153815d440 con 0x7f153806d560 2024-04-25T15:20:06.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 0x7f153815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 0x7f153815f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36716/0 (socket says 172.21.15.146:36716) 2024-04-25T15:20:06.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 -- 172.21.15.146:0/962811944 learned_addr learned my addr 172.21.15.146:0/962811944 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:06.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15367fc640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153815a610 0x7f153813de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f1536ffd640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153806d560 0x7f15381410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 -- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153806d560 msgr2=0x7f15381410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153806d560 0x7f15381410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 -- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153815a610 msgr2=0x7f153813de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153815a610 0x7f153813de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15377fe640 1 -- 172.21.15.146:0/962811944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153813eb80 con 0x7f153815d030 2024-04-25T15:20:06.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.503+0000 7f15367fc640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153815a610 0x7f153813de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:06.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.504+0000 7f15377fe640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 0x7f153815f1b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f153002ac90 tx=0x7f1530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.504+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153003f020 con 0x7f153815d030 2024-04-25T15:20:06.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.504+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530037040 con 0x7f153815d030 2024-04-25T15:20:06.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.504+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153813ee60 con 0x7f153815d030 2024-04-25T15:20:06.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.504+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530036070 con 0x7f153815d030 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.504+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153809c920 con 0x7f153815d030 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.505+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1538050740 con 0x7f153815d030 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.506+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1530042020 con 0x7f153815d030 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.507+0000 7f152f7fe640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15180744e0 0x7f15180769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.507+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f15300b5be0 con 0x7f153815d030 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.507+0000 7f1536ffd640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15180744e0 0x7f15180769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.507+0000 7f1536ffd640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15180744e0 0x7f15180769f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15380b42d0 tx=0x7f1528009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.512+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1530049020 con 0x7f153815d030 2024-04-25T15:20:06.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.695+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15380b6810 con 0x7f153815d030 2024-04-25T15:20:06.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.696+0000 7f152f7fe640 1 -- 172.21.15.146:0/962811944 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7f153007f9e0 con 0x7f153815d030 2024-04-25T15:20:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:06.697 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15180744e0 msgr2=0x7f15180769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15180744e0 0x7f15180769f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15380b42d0 tx=0x7f1528009040 comp rx=0 tx=0).stop 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 msgr2=0x7f153815f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 0x7f153815f1b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f153002ac90 tx=0x7f1530004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 shutdown_connections 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15180744e0 0x7f15180769f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f153815d030 0x7f153815f1b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153815a610 0x7f153813de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.698+0000 7f153d601640 1 --2- 172.21.15.146:0/962811944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153806d560 0x7f15381410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.699+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 >> 172.21.15.146:0/962811944 conn(0x7f15380731a0 msgr2=0x7f15380ae5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:06.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.699+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 shutdown_connections 2024-04-25T15:20:06.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.699+0000 7f153d601640 1 -- 172.21.15.146:0/962811944 wait complete. 2024-04-25T15:20:06.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:20:06.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.834+0000 7f964bc30640 1 Processor -- start 2024-04-25T15:20:06.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.835+0000 7f964bc30640 1 -- start start 2024-04-25T15:20:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f964bc30640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9644070b20 0x7f964406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f964bc30640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9644162350 0x7f9644157870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f964bc30640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964406deb0 con 0x7f964406daa0 2024-04-25T15:20:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f964bc30640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f964415ed80 con 0x7f9644070b20 2024-04-25T15:20:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f964bc30640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f964415e1f0 con 0x7f9644162350 2024-04-25T15:20:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9644070b20 0x7f964406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9644070b20 0x7f964406d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36728/0 (socket says 172.21.15.146:36728) 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 -- 172.21.15.146:0/3999572783 learned_addr learned my addr 172.21.15.146:0/3999572783 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96491cb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f964406daa0 0x7f9644161c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33950/0 (socket says 172.21.15.146:33950) 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f964a1cd640 1 --2- 172.21.15.146:0/3999572783 >> v2:172.21.15.177:3300/0 conn(0x7f9644162350 0x7f9644157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 -- 172.21.15.146:0/3999572783 >> v2:172.21.15.177:3300/0 conn(0x7f9644162350 msgr2=0x7f9644157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 --2- 172.21.15.146:0/3999572783 >> v2:172.21.15.177:3300/0 conn(0x7f9644162350 0x7f9644157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 -- 172.21.15.146:0/3999572783 >> v1:172.21.15.146:6789/0 conn(0x7f964406daa0 legacy=0x7f9644161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96489ca640 1 -- 172.21.15.146:0/3999572783 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825331839 0 0) 0x7f9634001110 con 0x7f964406daa0 2024-04-25T15:20:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.836+0000 7f96499cc640 1 -- 172.21.15.146:0/3999572783 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964415e450 con 0x7f9644070b20 2024-04-25T15:20:06.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.837+0000 7f96499cc640 1 --2- 172.21.15.146:0/3999572783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9644070b20 0x7f964406d560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9640009e50 tx=0x7f964002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5721a597e77805e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.837+0000 7f96489ca640 1 -- 172.21.15.146:0/3999572783 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640036070 con 0x7f9644070b20 2024-04-25T15:20:06.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.837+0000 7f96489ca640 1 -- 172.21.15.146:0/3999572783 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9640031600 con 0x7f9644070b20 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.837+0000 7f96489ca640 1 -- 172.21.15.146:0/3999572783 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96400319d0 con 0x7f9644070b20 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 -- 172.21.15.146:0/3999572783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9644070b20 msgr2=0x7f964406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 --2- 172.21.15.146:0/3999572783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9644070b20 0x7f964406d560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9640009e50 tx=0x7f964002a6d0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 -- 172.21.15.146:0/3999572783 shutdown_connections 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 --2- 172.21.15.146:0/3999572783 >> v2:172.21.15.177:3300/0 conn(0x7f9644162350 0x7f9644157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 --2- 172.21.15.146:0/3999572783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9644070b20 0x7f964406d560 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 -- 172.21.15.146:0/3999572783 >> 172.21.15.146:0/3999572783 conn(0x7f96440d1c10 msgr2=0x7f96440733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.838+0000 7f964bc30640 1 -- 172.21.15.146:0/3999572783 shutdown_connections 2024-04-25T15:20:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.839+0000 7f964bc30640 1 -- 172.21.15.146:0/3999572783 wait complete. 2024-04-25T15:20:06.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.840+0000 7f964bc30640 1 Processor -- start 2024-04-25T15:20:06.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.840+0000 7f964bc30640 1 -- start start 2024-04-25T15:20:06.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.841+0000 7f964bc30640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 0x7f96441568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.841+0000 7f964bc30640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9644070b20 0x7f96440b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.841+0000 7f964bc30640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9644162350 0x7f96440b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f964bc30640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9644070f30 con 0x7f964406daa0 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f964bc30640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f964415ea10 con 0x7f9644070b20 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f964bc30640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f96440710b0 con 0x7f9644162350 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 0x7f96441568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 0x7f96441568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36738/0 (socket says 172.21.15.146:36738) 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 -- 172.21.15.146:0/2081062087 learned_addr learned my addr 172.21.15.146:0/2081062087 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96491cb640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9644070b20 0x7f96440b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f964a1cd640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9644162350 0x7f96440b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 -- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9644162350 msgr2=0x7f96440b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9644162350 0x7f96440b2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 -- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9644070b20 msgr2=0x7f96440b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:06.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9644070b20 0x7f96440b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 -- 172.21.15.146:0/2081062087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96441f7900 con 0x7f964406daa0 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f96499cc640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 0x7f96441568f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9640009f70 tx=0x7f9640004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964003f020 con 0x7f964406daa0 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9640037040 con 0x7f964406daa0 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96441f7be0 con 0x7f964406daa0 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.842+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640036070 con 0x7f964406daa0 2024-04-25T15:20:06.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.843+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96441f81a0 con 0x7f964406daa0 2024-04-25T15:20:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.844+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964406a0e0 con 0x7f964406daa0 2024-04-25T15:20:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.844+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9640042020 con 0x7f964406daa0 2024-04-25T15:20:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.845+0000 7f963a7fc640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f96280744b0 0x7f96280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.845+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f96400b5910 con 0x7f964406daa0 2024-04-25T15:20:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.845+0000 7f96491cb640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f96280744b0 0x7f96280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.845+0000 7f96491cb640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f96280744b0 0x7f96280769c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f96441e7a20 tx=0x7f9634007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:06.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:06.850+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9640049030 con 0x7f964406daa0 2024-04-25T15:20:07.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.074+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9644073d50 con 0x7f964406daa0 2024-04-25T15:20:07.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.542+0000 7f963a7fc640 1 -- 172.21.15.146:0/2081062087 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f964007f7a0 con 0x7f964406daa0 2024-04-25T15:20:07.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f96280744b0 msgr2=0x7f96280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f96280744b0 0x7f96280769c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f96441e7a20 tx=0x7f9634007e40 comp rx=0 tx=0).stop 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 msgr2=0x7f96441568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 0x7f96441568f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9640009f70 tx=0x7f9640004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 shutdown_connections 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f96280744b0 0x7f96280769c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9644162350 0x7f96440b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9644070b20 0x7f96440b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 --2- 172.21.15.146:0/2081062087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f964406daa0 0x7f96441568f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 >> 172.21.15.146:0/2081062087 conn(0x7f96440d1c10 msgr2=0x7f9644161c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 shutdown_connections 2024-04-25T15:20:07.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.545+0000 7f964bc30640 1 -- 172.21.15.146:0/2081062087 wait complete. 2024-04-25T15:20:07.557 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-25T15:20:07.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:20:07.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.670+0000 7f808d35e640 1 Processor -- start 2024-04-25T15:20:07.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.670+0000 7f808d35e640 1 -- start start 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.671+0000 7f808d35e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f80880bb450 0x7f80880d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.671+0000 7f808d35e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80880b5230 con 0x7f8088163c00 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.671+0000 7f808d35e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f808806c8c0 con 0x7f80880d0de0 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.671+0000 7f808d35e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f808806cba0 con 0x7f80880bb450 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8086ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8088163c00 0x7f808806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33966/0 (socket says 172.21.15.146:33966) 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8086ffd640 1 -- 172.21.15.146:0/267573164 learned_addr learned my addr 172.21.15.146:0/267573164 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f80867fc640 1 --2- 172.21.15.146:0/267573164 >> v2:172.21.15.177:3300/0 conn(0x7f80880bb450 0x7f80880d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 354710848 0 0) 0x7f80880b5230 con 0x7f8088163c00 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8070003520 con 0x7f8088163c00 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067385422 0 0) 0x7f808806c8c0 con 0x7f80880d0de0 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80880b5230 con 0x7f80880d0de0 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1442412798 0 0) 0x7f8070003520 con 0x7f8088163c00 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 >> v2:172.21.15.177:3300/0 conn(0x7f80880bb450 msgr2=0x7f80880d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 --2- 172.21.15.146:0/267573164 >> v2:172.21.15.177:3300/0 conn(0x7f80880bb450 0x7f80880d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 >> v1:172.21.15.177:6789/0 conn(0x7f80880d0de0 legacy=0x7f80880b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808806ce80 con 0x7f8088163c00 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f80780019f0 con 0x7f8088163c00 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 >> v1:172.21.15.146:6789/0 conn(0x7f8088163c00 legacy=0x7f808806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8070003bc0 0x7f8070004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8070004800 0x7f8070004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f807000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f808806cba0 con 0x7f8070005320 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f807000c250 con 0x7f8070003bc0 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f807000c450 con 0x7f8070004800 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f80877fe640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f807000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f8086ffd640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8070003bc0 0x7f8070004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.672+0000 7f80867fc640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8070004800 0x7f8070004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 -- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8070004800 msgr2=0x7f8070004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8070004800 0x7f8070004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 -- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8070003bc0 msgr2=0x7f8070004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8070003bc0 0x7f8070004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 -- 172.21.15.146:0/267573164 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8070003520 con 0x7f8070005320 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 -- 172.21.15.146:0/267573164 reap_dead start 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f80877fe640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f807000b9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80800018a0 tx=0x7f8070005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3145e4a3026eadd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800014f0 con 0x7f8070005320 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80800052d0 con 0x7f8070005320 2024-04-25T15:20:07.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.673+0000 7f8085ffb640 1 -- 172.21.15.146:0/267573164 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808002ac80 con 0x7f8070005320 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 -- 172.21.15.146:0/267573164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 msgr2=0x7f807000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f807000b9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80800018a0 tx=0x7f8070005830 comp rx=0 tx=0).stop 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 -- 172.21.15.146:0/267573164 shutdown_connections 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 --2- 172.21.15.146:0/267573164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f807000b9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 -- 172.21.15.146:0/267573164 >> 172.21.15.146:0/267573164 conn(0x7f80880b7970 msgr2=0x7f80880b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 -- 172.21.15.146:0/267573164 shutdown_connections 2024-04-25T15:20:07.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.674+0000 7f808d35e640 1 -- 172.21.15.146:0/267573164 wait complete. 2024-04-25T15:20:07.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.676+0000 7f808d35e640 1 Processor -- start 2024-04-25T15:20:07.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.676+0000 7f808d35e640 1 -- start start 2024-04-25T15:20:07.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.677+0000 7f808d35e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f80881618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.677+0000 7f808d35e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80881f73a0 0x7f80881f92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.677+0000 7f808d35e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80881f6560 0x7f80881f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.677+0000 7f808d35e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f80880bb860 con 0x7f8070005320 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.677+0000 7f808d35e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f808806c550 con 0x7f80881f6560 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.677+0000 7f808d35e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f80880bb9e0 con 0x7f80881f73a0 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f80881618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f80881618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36758/0 (socket says 172.21.15.146:36758) 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 -- 172.21.15.146:0/4283191717 learned_addr learned my addr 172.21.15.146:0/4283191717 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f80867fc640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80881f73a0 0x7f80881f92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f80877fe640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80881f6560 0x7f80881f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 -- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80881f73a0 msgr2=0x7f80881f92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80881f73a0 0x7f80881f92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 -- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80881f6560 msgr2=0x7f80881f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80881f6560 0x7f80881f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:07.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 -- 172.21.15.146:0/4283191717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80881f9830 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f80877fe640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80881f6560 0x7f80881f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f8086ffd640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f80881618a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8078007770 tx=0x7f8078007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078017020 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f807800ab40 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80881620f0 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780133f0 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.678+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80881626b0 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.679+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808806a0e0 con 0x7f8070005320 2024-04-25T15:20:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.680+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f807801a020 con 0x7f8070005320 2024-04-25T15:20:07.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.680+0000 7f806f7fe640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80740744b0 0x7f80740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:07.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.680+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8078095cf0 con 0x7f8070005320 2024-04-25T15:20:07.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.680+0000 7f80867fc640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80740744b0 0x7f80740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:07.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.681+0000 7f80867fc640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80740744b0 0x7f80740769c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8088068810 tx=0x7f807c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:07.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.685+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f807809a070 con 0x7f8070005320 2024-04-25T15:20:07.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:07.899+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f80880d3ed0 con 0x7f8070005320 2024-04-25T15:20:08.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.544+0000 7f806f7fe640 1 -- 172.21.15.146:0/4283191717 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f8078021020 con 0x7f8070005320 2024-04-25T15:20:08.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80740744b0 msgr2=0x7f80740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80740744b0 0x7f80740769c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8088068810 tx=0x7f807c009190 comp rx=0 tx=0).stop 2024-04-25T15:20:08.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 msgr2=0x7f80881618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f80881618a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8078007770 tx=0x7f8078007b70 comp rx=0 tx=0).stop 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 shutdown_connections 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80740744b0 0x7f80740769c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80881f6560 0x7f80881f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80881f73a0 0x7f80881f92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 --2- 172.21.15.146:0/4283191717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8070005320 0x7f80881618a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 >> 172.21.15.146:0/4283191717 conn(0x7f80880b7970 msgr2=0x7f80880b9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 shutdown_connections 2024-04-25T15:20:08.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.546+0000 7f808d35e640 1 -- 172.21.15.146:0/4283191717 wait complete. 2024-04-25T15:20:08.559 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:20:08.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:08.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.668+0000 7f3f42da8640 1 Processor -- start 2024-04-25T15:20:08.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.668+0000 7f3f42da8640 1 -- start start 2024-04-25T15:20:08.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f42da8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f3c1691d0 0x7f3f3c16b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:08.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f42da8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3c0b8eb0 con 0x7f3f3c06ee60 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f42da8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3c08ce00 con 0x7f3f3c0b2ba0 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f42da8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3f3c161840 con 0x7f3f3c1691d0 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f41345640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f3c1691d0 0x7f3f3c16b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f40b44640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3f3c06ee60 0x7f3f3c163eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33978/0 (socket says 172.21.15.146:33978) 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.669+0000 7f3f40b44640 1 -- 172.21.15.146:0/3923912419 learned_addr learned my addr 172.21.15.146:0/3923912419 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.670+0000 7f3f41345640 1 -- 172.21.15.146:0/3923912419 >> v1:172.21.15.177:6789/0 conn(0x7f3f3c0b2ba0 legacy=0x7f3f3c1645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.670+0000 7f3f41345640 1 -- 172.21.15.146:0/3923912419 >> v1:172.21.15.146:6789/0 conn(0x7f3f3c06ee60 legacy=0x7f3f3c163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.670+0000 7f3f3b7fe640 1 -- 172.21.15.146:0/3923912419 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223486036 0 0) 0x7f3f3c0b8eb0 con 0x7f3f3c06ee60 2024-04-25T15:20:08.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.670+0000 7f3f41345640 1 -- 172.21.15.146:0/3923912419 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c08d0e0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.670+0000 7f3f41345640 1 --2- 172.21.15.146:0/3923912419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c16b5f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009d90 tx=0x7f3f3402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e3d7ac8d39f264 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:08.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f3b7fe640 1 -- 172.21.15.146:0/3923912419 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34007740 con 0x7f3f3c1691d0 2024-04-25T15:20:08.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f3b7fe640 1 -- 172.21.15.146:0/3923912419 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f340078e0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f3b7fe640 1 -- 172.21.15.146:0/3923912419 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34007cb0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 -- 172.21.15.146:0/3923912419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 msgr2=0x7f3f3c16b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 --2- 172.21.15.146:0/3923912419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c16b5f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009d90 tx=0x7f3f3402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 -- 172.21.15.146:0/3923912419 shutdown_connections 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 --2- 172.21.15.146:0/3923912419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c16b5f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 -- 172.21.15.146:0/3923912419 >> 172.21.15.146:0/3923912419 conn(0x7f3f3c0b52e0 msgr2=0x7f3f3c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 -- 172.21.15.146:0/3923912419 shutdown_connections 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.671+0000 7f3f42da8640 1 -- 172.21.15.146:0/3923912419 wait complete. 2024-04-25T15:20:08.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.673+0000 7f3f42da8640 1 Processor -- start 2024-04-25T15:20:08.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.673+0000 7f3f42da8640 1 -- start start 2024-04-25T15:20:08.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.674+0000 7f3f42da8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f3c06ee60 0x7f3f3c14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:08.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.674+0000 7f3f42da8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f3c0b2ba0 0x7f3f3c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:08.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.674+0000 7f3f42da8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c0b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:08.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.674+0000 7f3f42da8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c06f440 con 0x7f3f3c1691d0 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.674+0000 7f3f42da8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c08d730 con 0x7f3f3c06ee60 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.674+0000 7f3f42da8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3c08dbc0 con 0x7f3f3c0b2ba0 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c0b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c0b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36788/0 (socket says 172.21.15.146:36788) 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 -- 172.21.15.146:0/669861732 learned_addr learned my addr 172.21.15.146:0/669861732 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f40b44640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f3c06ee60 0x7f3f3c14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f3bfff640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f3c0b2ba0 0x7f3f3c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 -- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f3c0b2ba0 msgr2=0x7f3f3c0b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f3c0b2ba0 0x7f3f3c0b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 -- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f3c06ee60 msgr2=0x7f3f3c14fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f3c06ee60 0x7f3f3c14fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 -- 172.21.15.146:0/669861732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c163530 con 0x7f3f3c1691d0 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f40b44640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f3c06ee60 0x7f3f3c14fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:08.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f41345640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c0b0a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402ac90 tx=0x7f3f34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:08.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34040070 con 0x7f3f3c1691d0 2024-04-25T15:20:08.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f34031c40 con 0x7f3f3c1691d0 2024-04-25T15:20:08.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3c163810 con 0x7f3f3c1691d0 2024-04-25T15:20:08.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34003990 con 0x7f3f3c1691d0 2024-04-25T15:20:08.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.675+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3c09e960 con 0x7f3f3c1691d0 2024-04-25T15:20:08.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.676+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3c06a0e0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.677+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3f34039040 con 0x7f3f3c1691d0 2024-04-25T15:20:08.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.677+0000 7f3f397fa640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f200744e0 0x7f3f200769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:08.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.677+0000 7f3f40b44640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f200744e0 0x7f3f200769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:08.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.677+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3f340b6480 con 0x7f3f3c1691d0 2024-04-25T15:20:08.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.678+0000 7f3f40b44640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f200744e0 0x7f3f200769f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f300010e0 tx=0x7f3f30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:08.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.682+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f340808d0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.900+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f3c0b61f0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.900+0000 7f3f397fa640 1 -- 172.21.15.146:0/669861732 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7f3f340801d0 con 0x7f3f3c1691d0 2024-04-25T15:20:08.901 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:08.901 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:08.901 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:08.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f200744e0 msgr2=0x7f3f200769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f200744e0 0x7f3f200769f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f300010e0 tx=0x7f3f30009040 comp rx=0 tx=0).stop 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 msgr2=0x7f3f3c0b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c0b0a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402ac90 tx=0x7f3f34004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 shutdown_connections 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f200744e0 0x7f3f200769f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f3c1691d0 0x7f3f3c0b0a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f3c0b2ba0 0x7f3f3c0b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 --2- 172.21.15.146:0/669861732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f3c06ee60 0x7f3f3c14fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 >> 172.21.15.146:0/669861732 conn(0x7f3f3c0b52e0 msgr2=0x7f3f3c160600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.902+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 shutdown_connections 2024-04-25T15:20:08.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:08.903+0000 7f3f42da8640 1 -- 172.21.15.146:0/669861732 wait complete. 2024-04-25T15:20:08.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:09.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.021+0000 7fa20e614640 1 Processor -- start 2024-04-25T15:20:09.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.021+0000 7fa20e614640 1 -- start start 2024-04-25T15:20:09.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa20e614640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa208162890 0x7fa20808d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa20e614640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa2080b4df0 0x7fa208161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa20e614640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa208162520 con 0x7fa208162110 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa20e614640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa2080d23a0 con 0x7fa208162890 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa20e614640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa2080d1790 con 0x7fa2080b4df0 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa207fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa208162890 0x7fa20808d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa207fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa208162890 0x7fa20808d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36796/0 (socket says 172.21.15.146:36796) 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa207fff640 1 -- 172.21.15.146:0/2509403150 learned_addr learned my addr 172.21.15.146:0/2509403150 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.022+0000 7fa20cbb1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa208162110 0x7fa20815e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33988/0 (socket says 172.21.15.146:33988) 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa2077fe640 1 --2- 172.21.15.146:0/2509403150 >> v2:172.21.15.177:3300/0 conn(0x7fa2080b4df0 0x7fa208161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa207fff640 1 -- 172.21.15.146:0/2509403150 >> v2:172.21.15.177:3300/0 conn(0x7fa2080b4df0 msgr2=0x7fa208161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:09.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa207fff640 1 --2- 172.21.15.146:0/2509403150 >> v2:172.21.15.177:3300/0 conn(0x7fa2080b4df0 0x7fa208161bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa207fff640 1 -- 172.21.15.146:0/2509403150 >> v1:172.21.15.146:6789/0 conn(0x7fa208162110 legacy=0x7fa20815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa207fff640 1 -- 172.21.15.146:0/2509403150 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2080d1a70 con 0x7fa208162890 2024-04-25T15:20:09.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa206ffd640 1 -- 172.21.15.146:0/2509403150 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777286967 0 0) 0x7fa200001490 con 0x7fa208162110 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.023+0000 7fa207fff640 1 --2- 172.21.15.146:0/2509403150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa208162890 0x7fa20808d670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa2080d1370 tx=0x7fa1f002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8528832938d638fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa206ffd640 1 -- 172.21.15.146:0/2509403150 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0004030 con 0x7fa208162890 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa206ffd640 1 -- 172.21.15.146:0/2509403150 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f00041d0 con 0x7fa208162890 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa206ffd640 1 -- 172.21.15.146:0/2509403150 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0003610 con 0x7fa208162890 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 -- 172.21.15.146:0/2509403150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa208162890 msgr2=0x7fa20808d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 --2- 172.21.15.146:0/2509403150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa208162890 0x7fa20808d670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa2080d1370 tx=0x7fa1f002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 -- 172.21.15.146:0/2509403150 shutdown_connections 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 --2- 172.21.15.146:0/2509403150 >> v2:172.21.15.177:3300/0 conn(0x7fa2080b4df0 0x7fa208161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 --2- 172.21.15.146:0/2509403150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa208162890 0x7fa20808d670 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 -- 172.21.15.146:0/2509403150 >> 172.21.15.146:0/2509403150 conn(0x7fa2080b52e0 msgr2=0x7fa2080b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:09.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 -- 172.21.15.146:0/2509403150 shutdown_connections 2024-04-25T15:20:09.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.024+0000 7fa20e614640 1 -- 172.21.15.146:0/2509403150 wait complete. 2024-04-25T15:20:09.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.026+0000 7fa20e614640 1 Processor -- start 2024-04-25T15:20:09.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.026+0000 7fa20e614640 1 -- start start 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 0x7fa20809e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa208162110 0x7fa20815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa208162890 0x7fa2081597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa2080d2830 con 0x7fa2080b4df0 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa2080d2030 con 0x7fa208162110 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa20808dbb0 con 0x7fa208162890 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 0x7fa20809e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 0x7fa20809e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36798/0 (socket says 172.21.15.146:36798) 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 -- 172.21.15.146:0/3448251930 learned_addr learned my addr 172.21.15.146:0/3448251930 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa2077fe640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa208162110 0x7fa20815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20cbb1640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa208162890 0x7fa2081597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 -- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa208162890 msgr2=0x7fa2081597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa208162890 0x7fa2081597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 -- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa208162110 msgr2=0x7fa20815aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa208162110 0x7fa20815aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 -- 172.21.15.146:0/3448251930 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20809d920 con 0x7fa2080b4df0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa207fff640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 0x7fa20809e9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa1f002a200 tx=0x7fa1f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0041020 con 0x7fa2080b4df0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f0003c50 con 0x7fa2080b4df0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0004030 con 0x7fa2080b4df0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20809dc00 con 0x7fa2080b4df0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.028+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20809e1c0 con 0x7fa2080b4df0 2024-04-25T15:20:09.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.029+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa20806a0e0 con 0x7fa2080b4df0 2024-04-25T15:20:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.030+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa1f0044020 con 0x7fa2080b4df0 2024-04-25T15:20:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.030+0000 7fa204ff9640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1e80744b0 0x7fa1e80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.030+0000 7fa2077fe640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1e80744b0 0x7fa1e80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.030+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa1f00b6180 con 0x7fa2080b4df0 2024-04-25T15:20:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.031+0000 7fa2077fe640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1e80744b0 0x7fa1e80769c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa208072a30 tx=0x7fa1f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.035+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa1f004b030 con 0x7fa2080b4df0 2024-04-25T15:20:09.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.208+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2080b61f0 con 0x7fa2080b4df0 2024-04-25T15:20:09.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.209+0000 7fa204ff9640 1 -- 172.21.15.146:0/3448251930 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7fa1f007ff80 con 0x7fa2080b4df0 2024-04-25T15:20:09.211 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:09.211 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:09.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1e80744b0 msgr2=0x7fa1e80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1e80744b0 0x7fa1e80769c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa208072a30 tx=0x7fa1f8008040 comp rx=0 tx=0).stop 2024-04-25T15:20:09.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 msgr2=0x7fa20809e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 0x7fa20809e9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa1f002a200 tx=0x7fa1f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 shutdown_connections 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1e80744b0 0x7fa1e80769c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa208162890 0x7fa2081597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa208162110 0x7fa20815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 --2- 172.21.15.146:0/3448251930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2080b4df0 0x7fa20809e9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.211+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 >> 172.21.15.146:0/3448251930 conn(0x7fa2080b52e0 msgr2=0x7fa208072a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.212+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 shutdown_connections 2024-04-25T15:20:09.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.212+0000 7fa20e614640 1 -- 172.21.15.146:0/3448251930 wait complete. 2024-04-25T15:20:09.224 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:20:09.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:20:09.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.331+0000 7f41467c7640 1 Processor -- start 2024-04-25T15:20:09.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.331+0000 7f41467c7640 1 -- start start 2024-04-25T15:20:09.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f41467c7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f41400d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f41467c7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f41400b5950 0x7f41400b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f41467c7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41400d2230 con 0x7f414015ffa0 2024-04-25T15:20:09.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f41467c7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f41400b2f50 con 0x7f41400b5950 2024-04-25T15:20:09.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f41467c7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f41400b2ac0 con 0x7f41400b6610 2024-04-25T15:20:09.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f41400b5950 0x7f41400b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f41400b5950 0x7f41400b7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36816/0 (socket says 172.21.15.146:36816) 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 -- 172.21.15.146:0/1092910856 learned_addr learned my addr 172.21.15.146:0/1092910856 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f413ffff640 1 --2- 172.21.15.146:0/1092910856 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f41400d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 -- 172.21.15.146:0/1092910856 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 msgr2=0x7f41400d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 --2- 172.21.15.146:0/1092910856 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f41400d2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 -- 172.21.15.146:0/1092910856 >> v1:172.21.15.177:6789/0 conn(0x7f414015ffa0 legacy=0x7f41400b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.333+0000 7f4144d64640 1 -- 172.21.15.146:0/1092910856 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41400cfa80 con 0x7f41400b5950 2024-04-25T15:20:09.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.334+0000 7f4144d64640 1 --2- 172.21.15.146:0/1092910856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400b5950 0x7f41400b7cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4138009e50 tx=0x7f413802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5ac0d0e0e80769 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.334+0000 7f413effd640 1 -- 172.21.15.146:0/1092910856 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138036070 con 0x7f41400b5950 2024-04-25T15:20:09.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.334+0000 7f413effd640 1 -- 172.21.15.146:0/1092910856 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4138031600 con 0x7f41400b5950 2024-04-25T15:20:09.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.334+0000 7f413effd640 1 -- 172.21.15.146:0/1092910856 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41380319d0 con 0x7f41400b5950 2024-04-25T15:20:09.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 -- 172.21.15.146:0/1092910856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400b5950 msgr2=0x7f41400b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 --2- 172.21.15.146:0/1092910856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400b5950 0x7f41400b7cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4138009e50 tx=0x7f413802a710 comp rx=0 tx=0).stop 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 -- 172.21.15.146:0/1092910856 shutdown_connections 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 --2- 172.21.15.146:0/1092910856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400b5950 0x7f41400b7cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 --2- 172.21.15.146:0/1092910856 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f41400d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 -- 172.21.15.146:0/1092910856 >> 172.21.15.146:0/1092910856 conn(0x7f41400d0240 msgr2=0x7f41400d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 -- 172.21.15.146:0/1092910856 shutdown_connections 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.335+0000 7f41467c7640 1 -- 172.21.15.146:0/1092910856 wait complete. 2024-04-25T15:20:09.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.337+0000 7f41467c7640 1 Processor -- start 2024-04-25T15:20:09.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.337+0000 7f41467c7640 1 -- start start 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f41467c7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f4140154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f41467c7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f414015ffa0 0x7f41401f9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f41467c7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 0x7f41400c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f41467c7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f41400d1da0 con 0x7f41400c8680 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f41467c7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f41401603b0 con 0x7f414015ffa0 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f41467c7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f41400d1f20 con 0x7f41400b6610 2024-04-25T15:20:09.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f4144d64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 0x7f41400c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f4144d64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 0x7f41400c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36820/0 (socket says 172.21.15.146:36820) 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f4144d64640 1 -- 172.21.15.146:0/4079907671 learned_addr learned my addr 172.21.15.146:0/4079907671 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f413ffff640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f4140154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.338+0000 7f413f7fe640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f414015ffa0 0x7f41401f9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f4144d64640 1 -- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 msgr2=0x7f4140154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f4144d64640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f4140154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f4144d64640 1 -- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f414015ffa0 msgr2=0x7f41401f9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f4144d64640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f414015ffa0 0x7f41401f9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f4144d64640 1 -- 172.21.15.146:0/4079907671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41400c8fd0 con 0x7f41400c8680 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f4144d64640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 0x7f41400c8a90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4138034040 tx=0x7f4138004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413803f020 con 0x7f41400c8680 2024-04-25T15:20:09.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4138037040 con 0x7f41400c8680 2024-04-25T15:20:09.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138036070 con 0x7f41400c8680 2024-04-25T15:20:09.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41400ae970 con 0x7f41400c8680 2024-04-25T15:20:09.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.339+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41401fa590 con 0x7f41400c8680 2024-04-25T15:20:09.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.340+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f414006a0e0 con 0x7f41400c8680 2024-04-25T15:20:09.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.341+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4138042020 con 0x7f41400c8680 2024-04-25T15:20:09.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.341+0000 7f413cff9640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f412c0744e0 0x7f412c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.341+0000 7f413ffff640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f412c0744e0 0x7f412c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.341+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f41380b5790 con 0x7f41400c8680 2024-04-25T15:20:09.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.342+0000 7f413ffff640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f412c0744e0 0x7f412c0769f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4130009aa0 tx=0x7f4130009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.346+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4138049030 con 0x7f41400c8680 2024-04-25T15:20:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.519+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f41400b5800 con 0x7f412c0744e0 2024-04-25T15:20:09.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.527+0000 7f413cff9640 1 -- 172.21.15.146:0/4079907671 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7f41400b5800 con 0x7f412c0744e0 2024-04-25T15:20:09.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:20:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f412c0744e0 msgr2=0x7f412c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f412c0744e0 0x7f412c0769f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4130009aa0 tx=0x7f4130009190 comp rx=0 tx=0).stop 2024-04-25T15:20:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 msgr2=0x7f41400c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 0x7f41400c8a90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4138034040 tx=0x7f4138004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 shutdown_connections 2024-04-25T15:20:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f412c0744e0 0x7f412c0769f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41400c8680 0x7f41400c8a90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f414015ffa0 0x7f41401f9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 --2- 172.21.15.146:0/4079907671 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41400b6610 0x7f4140154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 >> 172.21.15.146:0/4079907671 conn(0x7f41400d0240 msgr2=0x7f41400b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 shutdown_connections 2024-04-25T15:20:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.531+0000 7f41467c7640 1 -- 172.21.15.146:0/4079907671 wait complete. 2024-04-25T15:20:09.543 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-25T15:20:07.991797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489335+0000","last_change":"2024-04-25T15:19:45.467889+0000","last_active":"2024-04-25T15:19:52.489335+0000","last_peered":"2024-04-25T15:19:52.489335+0000","last_clean":"2024-04-25T15:19:52.489335+0000","last_became_active":"2024-04-25T15:19:45.467565+0000","last_became_peered":"2024-04-25T15:19:45.467565+0000","last_unstale":"2024-04-25T15:19:52.489335+0000","last_undegraded":"2024-04-25T15:19:52.489335+0000","last_fullsized":"2024-04-25T15:19:52.489335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020856+0000","last_change":"2024-04-25T15:19:46.476186+0000","last_active":"2024-04-25T15:19:54.020856+0000","last_peered":"2024-04-25T15:19:54.020856+0000","last_clean":"2024-04-25T15:19:54.020856+0000","last_became_active":"2024-04-25T15:19:46.475865+0000","last_became_peered":"2024-04-25T15:19:46.475865+0000","last_unstale":"2024-04-25T15:19:54.020856+0000","last_undegraded":"2024-04-25T15:19:54.020856+0000","last_fullsized":"2024-04-25T15:19:54.020856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020377+0000","last_change":"2024-04-25T15:19:51.102406+0000","last_active":"2024-04-25T15:19:54.020377+0000","last_peered":"2024-04-25T15:19:54.020377+0000","last_clean":"2024-04-25T15:19:54.020377+0000","last_became_active":"2024-04-25T15:19:51.101849+0000","last_became_peered":"2024-04-25T15:19:51.101849+0000","last_unstale":"2024-04-25T15:19:54.020377+0000","last_undegraded":"2024-04-25T15:19:54.020377+0000","last_fullsized":"2024-04-25T15:19:54.020377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:58.329289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020474+0000","last_change":"2024-04-25T15:19:45.461077+0000","last_active":"2024-04-25T15:19:54.020474+0000","last_peered":"2024-04-25T15:19:54.020474+0000","last_clean":"2024-04-25T15:19:54.020474+0000","last_became_active":"2024-04-25T15:19:45.460662+0000","last_became_peered":"2024-04-25T15:19:45.460662+0000","last_unstale":"2024-04-25T15:19:54.020474+0000","last_undegraded":"2024-04-25T15:19:54.020474+0000","last_fullsized":"2024-04-25T15:19:54.020474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489205+0000","last_change":"2024-04-25T15:19:46.486683+0000","last_active":"2024-04-25T15:19:52.489205+0000","last_peered":"2024-04-25T15:19:52.489205+0000","last_clean":"2024-04-25T15:19:52.489205+0000","last_became_active":"2024-04-25T15:19:46.486203+0000","last_became_peered":"2024-04-25T15:19:46.486203+0000","last_unstale":"2024-04-25T15:19:52.489205+0000","last_undegraded":"2024-04-25T15:19:52.489205+0000","last_fullsized":"2024-04-25T15:19:52.489205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490288+0000","last_change":"2024-04-25T15:19:50.510692+0000","last_active":"2024-04-25T15:19:52.490288+0000","last_peered":"2024-04-25T15:19:52.490288+0000","last_clean":"2024-04-25T15:19:52.490288+0000","last_became_active":"2024-04-25T15:19:50.510156+0000","last_became_peered":"2024-04-25T15:19:50.510156+0000","last_unstale":"2024-04-25T15:19:52.490288+0000","last_undegraded":"2024-04-25T15:19:52.490288+0000","last_fullsized":"2024-04-25T15:19:52.490288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396965+0000","last_change":"2024-04-25T15:19:45.471053+0000","last_active":"2024-04-25T15:19:53.396965+0000","last_peered":"2024-04-25T15:19:53.396965+0000","last_clean":"2024-04-25T15:19:53.396965+0000","last_became_active":"2024-04-25T15:19:45.470704+0000","last_became_peered":"2024-04-25T15:19:45.470704+0000","last_unstale":"2024-04-25T15:19:53.396965+0000","last_undegraded":"2024-04-25T15:19:53.396965+0000","last_fullsized":"2024-04-25T15:19:53.396965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:55.012973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491359+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.491359+0000","last_peered":"2024-04-25T15:19:52.491359+0000","last_clean":"2024-04-25T15:19:52.491359+0000","last_became_active":"2024-04-25T15:19:46.482181+0000","last_became_peered":"2024-04-25T15:19:46.482181+0000","last_unstale":"2024-04-25T15:19:52.491359+0000","last_undegraded":"2024-04-25T15:19:52.491359+0000","last_fullsized":"2024-04-25T15:19:52.491359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492195+0000","last_change":"2024-04-25T15:19:50.515950+0000","last_active":"2024-04-25T15:19:52.492195+0000","last_peered":"2024-04-25T15:19:52.492195+0000","last_clean":"2024-04-25T15:19:52.492195+0000","last_became_active":"2024-04-25T15:19:50.515486+0000","last_became_peered":"2024-04-25T15:19:50.515486+0000","last_unstale":"2024-04-25T15:19:52.492195+0000","last_undegraded":"2024-04-25T15:19:52.492195+0000","last_fullsized":"2024-04-25T15:19:52.492195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396633+0000","last_change":"2024-04-25T15:19:45.472051+0000","last_active":"2024-04-25T15:19:53.396633+0000","last_peered":"2024-04-25T15:19:53.396633+0000","last_clean":"2024-04-25T15:19:53.396633+0000","last_became_active":"2024-04-25T15:19:45.471710+0000","last_became_peered":"2024-04-25T15:19:45.471710+0000","last_unstale":"2024-04-25T15:19:53.396633+0000","last_undegraded":"2024-04-25T15:19:53.396633+0000","last_fullsized":"2024-04-25T15:19:53.396633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491729+0000","last_change":"2024-04-25T15:19:46.483085+0000","last_active":"2024-04-25T15:19:52.491729+0000","last_peered":"2024-04-25T15:19:52.491729+0000","last_clean":"2024-04-25T15:19:52.491729+0000","last_became_active":"2024-04-25T15:19:46.482749+0000","last_became_peered":"2024-04-25T15:19:46.482749+0000","last_unstale":"2024-04-25T15:19:52.491729+0000","last_undegraded":"2024-04-25T15:19:52.491729+0000","last_fullsized":"2024-04-25T15:19:52.491729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490380+0000","last_change":"2024-04-25T15:19:50.508661+0000","last_active":"2024-04-25T15:19:52.490380+0000","last_peered":"2024-04-25T15:19:52.490380+0000","last_clean":"2024-04-25T15:19:52.490380+0000","last_became_active":"2024-04-25T15:19:50.508116+0000","last_became_peered":"2024-04-25T15:19:50.508116+0000","last_unstale":"2024-04-25T15:19:52.490380+0000","last_undegraded":"2024-04-25T15:19:52.490380+0000","last_fullsized":"2024-04-25T15:19:52.490380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018240+0000","last_change":"2024-04-25T15:19:46.477569+0000","last_active":"2024-04-25T15:19:54.018240+0000","last_peered":"2024-04-25T15:19:54.018240+0000","last_clean":"2024-04-25T15:19:54.018240+0000","last_became_active":"2024-04-25T15:19:46.477188+0000","last_became_peered":"2024-04-25T15:19:46.477188+0000","last_unstale":"2024-04-25T15:19:54.018240+0000","last_undegraded":"2024-04-25T15:19:54.018240+0000","last_fullsized":"2024-04-25T15:19:54.018240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396748+0000","last_change":"2024-04-25T15:19:45.469111+0000","last_active":"2024-04-25T15:19:53.396748+0000","last_peered":"2024-04-25T15:19:53.396748+0000","last_clean":"2024-04-25T15:19:53.396748+0000","last_became_active":"2024-04-25T15:19:45.468780+0000","last_became_peered":"2024-04-25T15:19:45.468780+0000","last_unstale":"2024-04-25T15:19:53.396748+0000","last_undegraded":"2024-04-25T15:19:53.396748+0000","last_fullsized":"2024-04-25T15:19:53.396748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491203+0000","last_change":"2024-04-25T15:19:50.513416+0000","last_active":"2024-04-25T15:19:52.491203+0000","last_peered":"2024-04-25T15:19:52.491203+0000","last_clean":"2024-04-25T15:19:52.491203+0000","last_became_active":"2024-04-25T15:19:50.513024+0000","last_became_peered":"2024-04-25T15:19:50.513024+0000","last_unstale":"2024-04-25T15:19:52.491203+0000","last_undegraded":"2024-04-25T15:19:52.491203+0000","last_fullsized":"2024-04-25T15:19:52.491203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:20.334921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020296+0000","last_change":"2024-04-25T15:19:46.472167+0000","last_active":"2024-04-25T15:19:54.020296+0000","last_peered":"2024-04-25T15:19:54.020296+0000","last_clean":"2024-04-25T15:19:54.020296+0000","last_became_active":"2024-04-25T15:19:46.471839+0000","last_became_peered":"2024-04-25T15:19:46.471839+0000","last_unstale":"2024-04-25T15:19:54.020296+0000","last_undegraded":"2024-04-25T15:19:54.020296+0000","last_fullsized":"2024-04-25T15:19:54.020296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809709+0000","last_change":"2024-04-25T15:19:45.467435+0000","last_active":"2024-04-25T15:19:53.809709+0000","last_peered":"2024-04-25T15:19:53.809709+0000","last_clean":"2024-04-25T15:19:53.809709+0000","last_became_active":"2024-04-25T15:19:45.466960+0000","last_became_peered":"2024-04-25T15:19:45.466960+0000","last_unstale":"2024-04-25T15:19:53.809709+0000","last_undegraded":"2024-04-25T15:19:53.809709+0000","last_fullsized":"2024-04-25T15:19:53.809709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491020+0000","last_change":"2024-04-25T15:19:50.515801+0000","last_active":"2024-04-25T15:19:52.491020+0000","last_peered":"2024-04-25T15:19:52.491020+0000","last_clean":"2024-04-25T15:19:52.491020+0000","last_became_active":"2024-04-25T15:19:50.515340+0000","last_became_peered":"2024-04-25T15:19:50.515340+0000","last_unstale":"2024-04-25T15:19:52.491020+0000","last_undegraded":"2024-04-25T15:19:52.491020+0000","last_fullsized":"2024-04-25T15:19:52.491020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:39.634911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020955+0000","last_change":"2024-04-25T15:19:46.470328+0000","last_active":"2024-04-25T15:19:54.020955+0000","last_peered":"2024-04-25T15:19:54.020955+0000","last_clean":"2024-04-25T15:19:54.020955+0000","last_became_active":"2024-04-25T15:19:46.469972+0000","last_became_peered":"2024-04-25T15:19:46.469972+0000","last_unstale":"2024-04-25T15:19:54.020955+0000","last_undegraded":"2024-04-25T15:19:54.020955+0000","last_fullsized":"2024-04-25T15:19:54.020955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.808963+0000","last_change":"2024-04-25T15:19:45.468754+0000","last_active":"2024-04-25T15:19:53.808963+0000","last_peered":"2024-04-25T15:19:53.808963+0000","last_clean":"2024-04-25T15:19:53.808963+0000","last_became_active":"2024-04-25T15:19:45.468374+0000","last_became_peered":"2024-04-25T15:19:45.468374+0000","last_unstale":"2024-04-25T15:19:53.808963+0000","last_undegraded":"2024-04-25T15:19:53.808963+0000","last_fullsized":"2024-04-25T15:19:53.808963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810131+0000","last_change":"2024-04-25T15:19:51.101484+0000","last_active":"2024-04-25T15:19:53.810131+0000","last_peered":"2024-04-25T15:19:53.810131+0000","last_clean":"2024-04-25T15:19:53.810131+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:53.810131+0000","last_undegraded":"2024-04-25T15:19:53.810131+0000","last_fullsized":"2024-04-25T15:19:53.810131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:57.500581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491979+0000","last_change":"2024-04-25T15:19:46.458762+0000","last_active":"2024-04-25T15:19:52.491979+0000","last_peered":"2024-04-25T15:19:52.491979+0000","last_clean":"2024-04-25T15:19:52.491979+0000","last_became_active":"2024-04-25T15:19:46.458438+0000","last_became_peered":"2024-04-25T15:19:46.458438+0000","last_unstale":"2024-04-25T15:19:52.491979+0000","last_undegraded":"2024-04-25T15:19:52.491979+0000","last_fullsized":"2024-04-25T15:19:52.491979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.402061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396261+0000","last_change":"2024-04-25T15:19:45.471672+0000","last_active":"2024-04-25T15:19:53.396261+0000","last_peered":"2024-04-25T15:19:53.396261+0000","last_clean":"2024-04-25T15:19:53.396261+0000","last_became_active":"2024-04-25T15:19:45.471095+0000","last_became_peered":"2024-04-25T15:19:45.471095+0000","last_unstale":"2024-04-25T15:19:53.396261+0000","last_undegraded":"2024-04-25T15:19:53.396261+0000","last_fullsized":"2024-04-25T15:19:53.396261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490630+0000","last_change":"2024-04-25T15:19:50.511923+0000","last_active":"2024-04-25T15:19:52.490630+0000","last_peered":"2024-04-25T15:19:52.490630+0000","last_clean":"2024-04-25T15:19:52.490630+0000","last_became_active":"2024-04-25T15:19:50.511350+0000","last_became_peered":"2024-04-25T15:19:50.511350+0000","last_unstale":"2024-04-25T15:19:52.490630+0000","last_undegraded":"2024-04-25T15:19:52.490630+0000","last_fullsized":"2024-04-25T15:19:52.490630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489343+0000","last_change":"2024-04-25T15:19:46.472562+0000","last_active":"2024-04-25T15:19:52.489343+0000","last_peered":"2024-04-25T15:19:52.489343+0000","last_clean":"2024-04-25T15:19:52.489343+0000","last_became_active":"2024-04-25T15:19:46.471397+0000","last_became_peered":"2024-04-25T15:19:46.471397+0000","last_unstale":"2024-04-25T15:19:52.489343+0000","last_undegraded":"2024-04-25T15:19:52.489343+0000","last_fullsized":"2024-04-25T15:19:52.489343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017764+0000","last_change":"2024-04-25T15:19:45.467859+0000","last_active":"2024-04-25T15:19:54.017764+0000","last_peered":"2024-04-25T15:19:54.017764+0000","last_clean":"2024-04-25T15:19:54.017764+0000","last_became_active":"2024-04-25T15:19:45.467461+0000","last_became_peered":"2024-04-25T15:19:45.467461+0000","last_unstale":"2024-04-25T15:19:54.017764+0000","last_undegraded":"2024-04-25T15:19:54.017764+0000","last_fullsized":"2024-04-25T15:19:54.017764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491639+0000","last_change":"2024-04-25T15:19:51.101973+0000","last_active":"2024-04-25T15:19:52.491639+0000","last_peered":"2024-04-25T15:19:52.491639+0000","last_clean":"2024-04-25T15:19:52.491639+0000","last_became_active":"2024-04-25T15:19:51.101046+0000","last_became_peered":"2024-04-25T15:19:51.101046+0000","last_unstale":"2024-04-25T15:19:52.491639+0000","last_undegraded":"2024-04-25T15:19:52.491639+0000","last_fullsized":"2024-04-25T15:19:52.491639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020585+0000","last_change":"2024-04-25T15:19:46.481579+0000","last_active":"2024-04-25T15:19:54.020585+0000","last_peered":"2024-04-25T15:19:54.020585+0000","last_clean":"2024-04-25T15:19:54.020585+0000","last_became_active":"2024-04-25T15:19:46.481265+0000","last_became_peered":"2024-04-25T15:19:46.481265+0000","last_unstale":"2024-04-25T15:19:54.020585+0000","last_undegraded":"2024-04-25T15:19:54.020585+0000","last_fullsized":"2024-04-25T15:19:54.020585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810014+0000","last_change":"2024-04-25T15:19:45.464607+0000","last_active":"2024-04-25T15:19:53.810014+0000","last_peered":"2024-04-25T15:19:53.810014+0000","last_clean":"2024-04-25T15:19:53.810014+0000","last_became_active":"2024-04-25T15:19:45.464278+0000","last_became_peered":"2024-04-25T15:19:45.464278+0000","last_unstale":"2024-04-25T15:19:53.810014+0000","last_undegraded":"2024-04-25T15:19:53.810014+0000","last_fullsized":"2024-04-25T15:19:53.810014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020107+0000","last_change":"2024-04-25T15:19:51.101501+0000","last_active":"2024-04-25T15:19:54.020107+0000","last_peered":"2024-04-25T15:19:54.020107+0000","last_clean":"2024-04-25T15:19:54.020107+0000","last_became_active":"2024-04-25T15:19:51.101130+0000","last_became_peered":"2024-04-25T15:19:51.101130+0000","last_unstale":"2024-04-25T15:19:54.020107+0000","last_undegraded":"2024-04-25T15:19:54.020107+0000","last_fullsized":"2024-04-25T15:19:54.020107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809775+0000","last_change":"2024-04-25T15:19:46.469600+0000","last_active":"2024-04-25T15:19:53.809775+0000","last_peered":"2024-04-25T15:19:53.809775+0000","last_clean":"2024-04-25T15:19:53.809775+0000","last_became_active":"2024-04-25T15:19:46.469254+0000","last_became_peered":"2024-04-25T15:19:46.469254+0000","last_unstale":"2024-04-25T15:19:53.809775+0000","last_undegraded":"2024-04-25T15:19:53.809775+0000","last_fullsized":"2024-04-25T15:19:53.809775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397107+0000","last_change":"2024-04-25T15:19:45.471728+0000","last_active":"2024-04-25T15:19:53.397107+0000","last_peered":"2024-04-25T15:19:53.397107+0000","last_clean":"2024-04-25T15:19:53.397107+0000","last_became_active":"2024-04-25T15:19:45.471426+0000","last_became_peered":"2024-04-25T15:19:45.471426+0000","last_unstale":"2024-04-25T15:19:53.397107+0000","last_undegraded":"2024-04-25T15:19:53.397107+0000","last_fullsized":"2024-04-25T15:19:53.397107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:39:45.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488880+0000","last_change":"2024-04-25T15:19:51.101520+0000","last_active":"2024-04-25T15:19:52.488880+0000","last_peered":"2024-04-25T15:19:52.488880+0000","last_clean":"2024-04-25T15:19:52.488880+0000","last_became_active":"2024-04-25T15:19:51.100989+0000","last_became_peered":"2024-04-25T15:19:51.100989+0000","last_unstale":"2024-04-25T15:19:52.488880+0000","last_undegraded":"2024-04-25T15:19:52.488880+0000","last_fullsized":"2024-04-25T15:19:52.488880+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491850+0000","last_change":"2024-04-25T15:19:46.482591+0000","last_active":"2024-04-25T15:19:52.491850+0000","last_peered":"2024-04-25T15:19:52.491850+0000","last_clean":"2024-04-25T15:19:52.491850+0000","last_became_active":"2024-04-25T15:19:46.482265+0000","last_became_peered":"2024-04-25T15:19:46.482265+0000","last_unstale":"2024-04-25T15:19:52.491850+0000","last_undegraded":"2024-04-25T15:19:52.491850+0000","last_fullsized":"2024-04-25T15:19:52.491850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809000+0000","last_change":"2024-04-25T15:19:45.470984+0000","last_active":"2024-04-25T15:19:53.809000+0000","last_peered":"2024-04-25T15:19:53.809000+0000","last_clean":"2024-04-25T15:19:53.809000+0000","last_became_active":"2024-04-25T15:19:45.470665+0000","last_became_peered":"2024-04-25T15:19:45.470665+0000","last_unstale":"2024-04-25T15:19:53.809000+0000","last_undegraded":"2024-04-25T15:19:53.809000+0000","last_fullsized":"2024-04-25T15:19:53.809000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809497+0000","last_change":"2024-04-25T15:19:50.509138+0000","last_active":"2024-04-25T15:19:53.809497+0000","last_peered":"2024-04-25T15:19:53.809497+0000","last_clean":"2024-04-25T15:19:53.809497+0000","last_became_active":"2024-04-25T15:19:50.508713+0000","last_became_peered":"2024-04-25T15:19:50.508713+0000","last_unstale":"2024-04-25T15:19:53.809497+0000","last_undegraded":"2024-04-25T15:19:53.809497+0000","last_fullsized":"2024-04-25T15:19:53.809497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492266+0000","last_change":"2024-04-25T15:19:46.459065+0000","last_active":"2024-04-25T15:19:52.492266+0000","last_peered":"2024-04-25T15:19:52.492266+0000","last_clean":"2024-04-25T15:19:52.492266+0000","last_became_active":"2024-04-25T15:19:46.458708+0000","last_became_peered":"2024-04-25T15:19:46.458708+0000","last_unstale":"2024-04-25T15:19:52.492266+0000","last_undegraded":"2024-04-25T15:19:52.492266+0000","last_fullsized":"2024-04-25T15:19:52.492266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:16.345566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491129+0000","last_change":"2024-04-25T15:19:45.464370+0000","last_active":"2024-04-25T15:19:52.491129+0000","last_peered":"2024-04-25T15:19:52.491129+0000","last_clean":"2024-04-25T15:19:52.491129+0000","last_became_active":"2024-04-25T15:19:45.464041+0000","last_became_peered":"2024-04-25T15:19:45.464041+0000","last_unstale":"2024-04-25T15:19:52.491129+0000","last_undegraded":"2024-04-25T15:19:52.491129+0000","last_fullsized":"2024-04-25T15:19:52.491129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809793+0000","last_change":"2024-04-25T15:19:50.516748+0000","last_active":"2024-04-25T15:19:53.809793+0000","last_peered":"2024-04-25T15:19:53.809793+0000","last_clean":"2024-04-25T15:19:53.809793+0000","last_became_active":"2024-04-25T15:19:50.515480+0000","last_became_peered":"2024-04-25T15:19:50.515480+0000","last_unstale":"2024-04-25T15:19:53.809793+0000","last_undegraded":"2024-04-25T15:19:53.809793+0000","last_fullsized":"2024-04-25T15:19:53.809793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:54.507021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809413+0000","last_change":"2024-04-25T15:19:46.469618+0000","last_active":"2024-04-25T15:19:53.809413+0000","last_peered":"2024-04-25T15:19:53.809413+0000","last_clean":"2024-04-25T15:19:53.809413+0000","last_became_active":"2024-04-25T15:19:46.469289+0000","last_became_peered":"2024-04-25T15:19:46.469289+0000","last_unstale":"2024-04-25T15:19:53.809413+0000","last_undegraded":"2024-04-25T15:19:53.809413+0000","last_fullsized":"2024-04-25T15:19:53.809413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020301+0000","last_change":"2024-04-25T15:19:45.462951+0000","last_active":"2024-04-25T15:19:54.020301+0000","last_peered":"2024-04-25T15:19:54.020301+0000","last_clean":"2024-04-25T15:19:54.020301+0000","last_became_active":"2024-04-25T15:19:45.462424+0000","last_became_peered":"2024-04-25T15:19:45.462424+0000","last_unstale":"2024-04-25T15:19:54.020301+0000","last_undegraded":"2024-04-25T15:19:54.020301+0000","last_fullsized":"2024-04-25T15:19:54.020301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810177+0000","last_change":"2024-04-25T15:19:50.519794+0000","last_active":"2024-04-25T15:19:53.810177+0000","last_peered":"2024-04-25T15:19:53.810177+0000","last_clean":"2024-04-25T15:19:53.810177+0000","last_became_active":"2024-04-25T15:19:50.519278+0000","last_became_peered":"2024-04-25T15:19:50.519278+0000","last_unstale":"2024-04-25T15:19:53.810177+0000","last_undegraded":"2024-04-25T15:19:53.810177+0000","last_fullsized":"2024-04-25T15:19:53.810177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488686+0000","last_change":"2024-04-25T15:19:46.459839+0000","last_active":"2024-04-25T15:19:52.488686+0000","last_peered":"2024-04-25T15:19:52.488686+0000","last_clean":"2024-04-25T15:19:52.488686+0000","last_became_active":"2024-04-25T15:19:46.459401+0000","last_became_peered":"2024-04-25T15:19:46.459401+0000","last_unstale":"2024-04-25T15:19:52.488686+0000","last_undegraded":"2024-04-25T15:19:52.488686+0000","last_fullsized":"2024-04-25T15:19:52.488686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:46.303899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396291+0000","last_change":"2024-04-25T15:19:45.471412+0000","last_active":"2024-04-25T15:19:53.396291+0000","last_peered":"2024-04-25T15:19:53.396291+0000","last_clean":"2024-04-25T15:19:53.396291+0000","last_became_active":"2024-04-25T15:19:45.470632+0000","last_became_peered":"2024-04-25T15:19:45.470632+0000","last_unstale":"2024-04-25T15:19:53.396291+0000","last_undegraded":"2024-04-25T15:19:53.396291+0000","last_fullsized":"2024-04-25T15:19:53.396291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489312+0000","last_change":"2024-04-25T15:19:51.101653+0000","last_active":"2024-04-25T15:19:52.489312+0000","last_peered":"2024-04-25T15:19:52.489312+0000","last_clean":"2024-04-25T15:19:52.489312+0000","last_became_active":"2024-04-25T15:19:51.101246+0000","last_became_peered":"2024-04-25T15:19:51.101246+0000","last_unstale":"2024-04-25T15:19:52.489312+0000","last_undegraded":"2024-04-25T15:19:52.489312+0000","last_fullsized":"2024-04-25T15:19:52.489312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020754+0000","last_change":"2024-04-25T15:19:46.488866+0000","last_active":"2024-04-25T15:19:54.020754+0000","last_peered":"2024-04-25T15:19:54.020754+0000","last_clean":"2024-04-25T15:19:54.020754+0000","last_became_active":"2024-04-25T15:19:46.488494+0000","last_became_peered":"2024-04-25T15:19:46.488494+0000","last_unstale":"2024-04-25T15:19:54.020754+0000","last_undegraded":"2024-04-25T15:19:54.020754+0000","last_fullsized":"2024-04-25T15:19:54.020754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490534+0000","last_change":"2024-04-25T15:19:45.459379+0000","last_active":"2024-04-25T15:19:52.490534+0000","last_peered":"2024-04-25T15:19:52.490534+0000","last_clean":"2024-04-25T15:19:52.490534+0000","last_became_active":"2024-04-25T15:19:45.458985+0000","last_became_peered":"2024-04-25T15:19:45.458985+0000","last_unstale":"2024-04-25T15:19:52.490534+0000","last_undegraded":"2024-04-25T15:19:52.490534+0000","last_fullsized":"2024-04-25T15:19:52.490534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396508+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:19:53.396508+0000","last_peered":"2024-04-25T15:19:53.396508+0000","last_clean":"2024-04-25T15:19:53.396508+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:19:53.396508+0000","last_undegraded":"2024-04-25T15:19:53.396508+0000","last_fullsized":"2024-04-25T15:19:53.396508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018620+0000","last_change":"2024-04-25T15:19:51.102362+0000","last_active":"2024-04-25T15:19:54.018620+0000","last_peered":"2024-04-25T15:19:54.018620+0000","last_clean":"2024-04-25T15:19:54.018620+0000","last_became_active":"2024-04-25T15:19:51.101137+0000","last_became_peered":"2024-04-25T15:19:51.101137+0000","last_unstale":"2024-04-25T15:19:54.018620+0000","last_undegraded":"2024-04-25T15:19:54.018620+0000","last_fullsized":"2024-04-25T15:19:54.018620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396808+0000","last_change":"2024-04-25T15:19:46.486533+0000","last_active":"2024-04-25T15:19:53.396808+0000","last_peered":"2024-04-25T15:19:53.396808+0000","last_clean":"2024-04-25T15:19:53.396808+0000","last_became_active":"2024-04-25T15:19:46.485775+0000","last_became_peered":"2024-04-25T15:19:46.485775+0000","last_unstale":"2024-04-25T15:19:53.396808+0000","last_undegraded":"2024-04-25T15:19:53.396808+0000","last_fullsized":"2024-04-25T15:19:53.396808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019912+0000","last_change":"2024-04-25T15:19:45.467231+0000","last_active":"2024-04-25T15:19:54.019912+0000","last_peered":"2024-04-25T15:19:54.019912+0000","last_clean":"2024-04-25T15:19:54.019912+0000","last_became_active":"2024-04-25T15:19:45.466874+0000","last_became_peered":"2024-04-25T15:19:45.466874+0000","last_unstale":"2024-04-25T15:19:54.019912+0000","last_undegraded":"2024-04-25T15:19:54.019912+0000","last_fullsized":"2024-04-25T15:19:54.019912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018302+0000","last_change":"2024-04-25T15:19:51.101630+0000","last_active":"2024-04-25T15:19:54.018302+0000","last_peered":"2024-04-25T15:19:54.018302+0000","last_clean":"2024-04-25T15:19:54.018302+0000","last_became_active":"2024-04-25T15:19:51.101121+0000","last_became_peered":"2024-04-25T15:19:51.101121+0000","last_unstale":"2024-04-25T15:19:54.018302+0000","last_undegraded":"2024-04-25T15:19:54.018302+0000","last_fullsized":"2024-04-25T15:19:54.018302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491492+0000","last_change":"2024-04-25T15:19:46.483183+0000","last_active":"2024-04-25T15:19:52.491492+0000","last_peered":"2024-04-25T15:19:52.491492+0000","last_clean":"2024-04-25T15:19:52.491492+0000","last_became_active":"2024-04-25T15:19:46.482618+0000","last_became_peered":"2024-04-25T15:19:46.482618+0000","last_unstale":"2024-04-25T15:19:52.491492+0000","last_undegraded":"2024-04-25T15:19:52.491492+0000","last_fullsized":"2024-04-25T15:19:52.491492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809024+0000","last_change":"2024-04-25T15:19:45.469018+0000","last_active":"2024-04-25T15:19:53.809024+0000","last_peered":"2024-04-25T15:19:53.809024+0000","last_clean":"2024-04-25T15:19:53.809024+0000","last_became_active":"2024-04-25T15:19:45.468712+0000","last_became_peered":"2024-04-25T15:19:45.468712+0000","last_unstale":"2024-04-25T15:19:53.809024+0000","last_undegraded":"2024-04-25T15:19:53.809024+0000","last_fullsized":"2024-04-25T15:19:53.809024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489037+0000","last_change":"2024-04-25T15:19:50.513573+0000","last_active":"2024-04-25T15:19:52.489037+0000","last_peered":"2024-04-25T15:19:52.489037+0000","last_clean":"2024-04-25T15:19:52.489037+0000","last_became_active":"2024-04-25T15:19:50.513018+0000","last_became_peered":"2024-04-25T15:19:50.513018+0000","last_unstale":"2024-04-25T15:19:52.489037+0000","last_undegraded":"2024-04-25T15:19:52.489037+0000","last_fullsized":"2024-04-25T15:19:52.489037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397172+0000","last_change":"2024-04-25T15:19:46.465000+0000","last_active":"2024-04-25T15:19:53.397172+0000","last_peered":"2024-04-25T15:19:53.397172+0000","last_clean":"2024-04-25T15:19:53.397172+0000","last_became_active":"2024-04-25T15:19:46.464680+0000","last_became_peered":"2024-04-25T15:19:46.464680+0000","last_unstale":"2024-04-25T15:19:53.397172+0000","last_undegraded":"2024-04-25T15:19:53.397172+0000","last_fullsized":"2024-04-25T15:19:53.397172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:35.597757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020764+0000","last_change":"2024-04-25T15:19:45.463067+0000","last_active":"2024-04-25T15:19:54.020764+0000","last_peered":"2024-04-25T15:19:54.020764+0000","last_clean":"2024-04-25T15:19:54.020764+0000","last_became_active":"2024-04-25T15:19:45.462611+0000","last_became_peered":"2024-04-25T15:19:45.462611+0000","last_unstale":"2024-04-25T15:19:54.020764+0000","last_undegraded":"2024-04-25T15:19:54.020764+0000","last_fullsized":"2024-04-25T15:19:54.020764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490099+0000","last_change":"2024-04-25T15:19:51.101470+0000","last_active":"2024-04-25T15:19:52.490099+0000","last_peered":"2024-04-25T15:19:52.490099+0000","last_clean":"2024-04-25T15:19:52.490099+0000","last_became_active":"2024-04-25T15:19:51.100991+0000","last_became_peered":"2024-04-25T15:19:51.100991+0000","last_unstale":"2024-04-25T15:19:52.490099+0000","last_undegraded":"2024-04-25T15:19:52.490099+0000","last_fullsized":"2024-04-25T15:19:52.490099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396886+0000","last_change":"2024-04-25T15:19:46.478228+0000","last_active":"2024-04-25T15:19:53.396886+0000","last_peered":"2024-04-25T15:19:53.396886+0000","last_clean":"2024-04-25T15:19:53.396886+0000","last_became_active":"2024-04-25T15:19:46.477939+0000","last_became_peered":"2024-04-25T15:19:46.477939+0000","last_unstale":"2024-04-25T15:19:53.396886+0000","last_undegraded":"2024-04-25T15:19:53.396886+0000","last_fullsized":"2024-04-25T15:19:53.396886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491269+0000","last_change":"2024-04-25T15:19:45.456572+0000","last_active":"2024-04-25T15:19:52.491269+0000","last_peered":"2024-04-25T15:19:52.491269+0000","last_clean":"2024-04-25T15:19:52.491269+0000","last_became_active":"2024-04-25T15:19:45.456238+0000","last_became_peered":"2024-04-25T15:19:45.456238+0000","last_unstale":"2024-04-25T15:19:52.491269+0000","last_undegraded":"2024-04-25T15:19:52.491269+0000","last_fullsized":"2024-04-25T15:19:52.491269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:09.326979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491720+0000","last_change":"2024-04-25T15:19:50.511879+0000","last_active":"2024-04-25T15:19:52.491720+0000","last_peered":"2024-04-25T15:19:52.491720+0000","last_clean":"2024-04-25T15:19:52.491720+0000","last_became_active":"2024-04-25T15:19:50.511380+0000","last_became_peered":"2024-04-25T15:19:50.511380+0000","last_unstale":"2024-04-25T15:19:52.491720+0000","last_undegraded":"2024-04-25T15:19:52.491720+0000","last_fullsized":"2024-04-25T15:19:52.491720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396446+0000","last_change":"2024-04-25T15:19:46.465026+0000","last_active":"2024-04-25T15:19:53.396446+0000","last_peered":"2024-04-25T15:19:53.396446+0000","last_clean":"2024-04-25T15:19:53.396446+0000","last_became_active":"2024-04-25T15:19:46.464703+0000","last_became_peered":"2024-04-25T15:19:46.464703+0000","last_unstale":"2024-04-25T15:19:53.396446+0000","last_undegraded":"2024-04-25T15:19:53.396446+0000","last_fullsized":"2024-04-25T15:19:53.396446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020165+0000","last_change":"2024-04-25T15:19:45.465017+0000","last_active":"2024-04-25T15:19:54.020165+0000","last_peered":"2024-04-25T15:19:54.020165+0000","last_clean":"2024-04-25T15:19:54.020165+0000","last_became_active":"2024-04-25T15:19:45.464691+0000","last_became_peered":"2024-04-25T15:19:45.464691+0000","last_unstale":"2024-04-25T15:19:54.020165+0000","last_undegraded":"2024-04-25T15:19:54.020165+0000","last_fullsized":"2024-04-25T15:19:54.020165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489222+0000","last_change":"2024-04-25T15:19:50.508942+0000","last_active":"2024-04-25T15:19:52.489222+0000","last_peered":"2024-04-25T15:19:52.489222+0000","last_clean":"2024-04-25T15:19:52.489222+0000","last_became_active":"2024-04-25T15:19:50.508482+0000","last_became_peered":"2024-04-25T15:19:50.508482+0000","last_unstale":"2024-04-25T15:19:52.489222+0000","last_undegraded":"2024-04-25T15:19:52.489222+0000","last_fullsized":"2024-04-25T15:19:52.489222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018243+0000","last_change":"2024-04-25T15:19:46.473708+0000","last_active":"2024-04-25T15:19:54.018243+0000","last_peered":"2024-04-25T15:19:54.018243+0000","last_clean":"2024-04-25T15:19:54.018243+0000","last_became_active":"2024-04-25T15:19:46.473372+0000","last_became_peered":"2024-04-25T15:19:46.473372+0000","last_unstale":"2024-04-25T15:19:54.018243+0000","last_undegraded":"2024-04-25T15:19:54.018243+0000","last_fullsized":"2024-04-25T15:19:54.018243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.469745+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.469458+0000","last_became_peered":"2024-04-25T15:19:45.469458+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020364+0000","last_change":"2024-04-25T15:19:51.101455+0000","last_active":"2024-04-25T15:19:54.020364+0000","last_peered":"2024-04-25T15:19:54.020364+0000","last_clean":"2024-04-25T15:19:54.020364+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:54.020364+0000","last_undegraded":"2024-04-25T15:19:54.020364+0000","last_fullsized":"2024-04-25T15:19:54.020364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:05.089655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396638+0000","last_change":"2024-04-25T15:19:46.465773+0000","last_active":"2024-04-25T15:19:53.396638+0000","last_peered":"2024-04-25T15:19:53.396638+0000","last_clean":"2024-04-25T15:19:53.396638+0000","last_became_active":"2024-04-25T15:19:46.465467+0000","last_became_peered":"2024-04-25T15:19:46.465467+0000","last_unstale":"2024-04-25T15:19:53.396638+0000","last_undegraded":"2024-04-25T15:19:53.396638+0000","last_fullsized":"2024-04-25T15:19:53.396638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491007+0000","last_change":"2024-04-25T15:19:45.463534+0000","last_active":"2024-04-25T15:19:52.491007+0000","last_peered":"2024-04-25T15:19:52.491007+0000","last_clean":"2024-04-25T15:19:52.491007+0000","last_became_active":"2024-04-25T15:19:45.463167+0000","last_became_peered":"2024-04-25T15:19:45.463167+0000","last_unstale":"2024-04-25T15:19:52.491007+0000","last_undegraded":"2024-04-25T15:19:52.491007+0000","last_fullsized":"2024-04-25T15:19:52.491007+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021379+0000","last_change":"2024-04-25T15:19:51.102717+0000","last_active":"2024-04-25T15:19:54.021379+0000","last_peered":"2024-04-25T15:19:54.021379+0000","last_clean":"2024-04-25T15:19:54.021379+0000","last_became_active":"2024-04-25T15:19:51.102247+0000","last_became_peered":"2024-04-25T15:19:51.102247+0000","last_unstale":"2024-04-25T15:19:54.021379+0000","last_undegraded":"2024-04-25T15:19:54.021379+0000","last_fullsized":"2024-04-25T15:19:54.021379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:14:00.372742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489038+0000","last_change":"2024-04-25T15:19:46.460776+0000","last_active":"2024-04-25T15:19:52.489038+0000","last_peered":"2024-04-25T15:19:52.489038+0000","last_clean":"2024-04-25T15:19:52.489038+0000","last_became_active":"2024-04-25T15:19:46.460453+0000","last_became_peered":"2024-04-25T15:19:46.460453+0000","last_unstale":"2024-04-25T15:19:52.489038+0000","last_undegraded":"2024-04-25T15:19:52.489038+0000","last_fullsized":"2024-04-25T15:19:52.489038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489102+0000","last_change":"2024-04-25T15:19:45.462749+0000","last_active":"2024-04-25T15:19:52.489102+0000","last_peered":"2024-04-25T15:19:52.489102+0000","last_clean":"2024-04-25T15:19:52.489102+0000","last_became_active":"2024-04-25T15:19:45.462417+0000","last_became_peered":"2024-04-25T15:19:45.462417+0000","last_unstale":"2024-04-25T15:19:52.489102+0000","last_undegraded":"2024-04-25T15:19:52.489102+0000","last_fullsized":"2024-04-25T15:19:52.489102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491884+0000","last_change":"2024-04-25T15:19:51.102630+0000","last_active":"2024-04-25T15:19:52.491884+0000","last_peered":"2024-04-25T15:19:52.491884+0000","last_clean":"2024-04-25T15:19:52.491884+0000","last_became_active":"2024-04-25T15:19:51.101912+0000","last_became_peered":"2024-04-25T15:19:51.101912+0000","last_unstale":"2024-04-25T15:19:52.491884+0000","last_undegraded":"2024-04-25T15:19:52.491884+0000","last_fullsized":"2024-04-25T15:19:52.491884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396712+0000","last_change":"2024-04-25T15:19:46.471583+0000","last_active":"2024-04-25T15:19:53.396712+0000","last_peered":"2024-04-25T15:19:53.396712+0000","last_clean":"2024-04-25T15:19:53.396712+0000","last_became_active":"2024-04-25T15:19:46.471286+0000","last_became_peered":"2024-04-25T15:19:46.471286+0000","last_unstale":"2024-04-25T15:19:53.396712+0000","last_undegraded":"2024-04-25T15:19:53.396712+0000","last_fullsized":"2024-04-25T15:19:53.396712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:55.948605+0000","last_change":"2024-04-25T15:19:45.464680+0000","last_active":"2024-04-25T15:19:55.948605+0000","last_peered":"2024-04-25T15:19:55.948605+0000","last_clean":"2024-04-25T15:19:55.948605+0000","last_became_active":"2024-04-25T15:19:45.463690+0000","last_became_peered":"2024-04-25T15:19:45.463690+0000","last_unstale":"2024-04-25T15:19:55.948605+0000","last_undegraded":"2024-04-25T15:19:55.948605+0000","last_fullsized":"2024-04-25T15:19:55.948605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809637+0000","last_change":"2024-04-25T15:19:50.509907+0000","last_active":"2024-04-25T15:19:53.809637+0000","last_peered":"2024-04-25T15:19:53.809637+0000","last_clean":"2024-04-25T15:19:53.809637+0000","last_became_active":"2024-04-25T15:19:50.508793+0000","last_became_peered":"2024-04-25T15:19:50.508793+0000","last_unstale":"2024-04-25T15:19:53.809637+0000","last_undegraded":"2024-04-25T15:19:53.809637+0000","last_fullsized":"2024-04-25T15:19:53.809637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490746+0000","last_change":"2024-04-25T15:19:46.471815+0000","last_active":"2024-04-25T15:19:52.490746+0000","last_peered":"2024-04-25T15:19:52.490746+0000","last_clean":"2024-04-25T15:19:52.490746+0000","last_became_active":"2024-04-25T15:19:46.471473+0000","last_became_peered":"2024-04-25T15:19:46.471473+0000","last_unstale":"2024-04-25T15:19:52.490746+0000","last_undegraded":"2024-04-25T15:19:52.490746+0000","last_fullsized":"2024-04-25T15:19:52.490746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809045+0000","last_change":"2024-04-25T15:19:45.472821+0000","last_active":"2024-04-25T15:19:53.809045+0000","last_peered":"2024-04-25T15:19:53.809045+0000","last_clean":"2024-04-25T15:19:53.809045+0000","last_became_active":"2024-04-25T15:19:45.472420+0000","last_became_peered":"2024-04-25T15:19:45.472420+0000","last_unstale":"2024-04-25T15:19:53.809045+0000","last_undegraded":"2024-04-25T15:19:53.809045+0000","last_fullsized":"2024-04-25T15:19:53.809045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492136+0000","last_change":"2024-04-25T15:19:50.509990+0000","last_active":"2024-04-25T15:19:52.492136+0000","last_peered":"2024-04-25T15:19:52.492136+0000","last_clean":"2024-04-25T15:19:52.492136+0000","last_became_active":"2024-04-25T15:19:50.509340+0000","last_became_peered":"2024-04-25T15:19:50.509340+0000","last_unstale":"2024-04-25T15:19:52.492136+0000","last_undegraded":"2024-04-25T15:19:52.492136+0000","last_fullsized":"2024-04-25T15:19:52.492136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489275+0000","last_change":"2024-04-25T15:19:46.485819+0000","last_active":"2024-04-25T15:19:52.489275+0000","last_peered":"2024-04-25T15:19:52.489275+0000","last_clean":"2024-04-25T15:19:52.489275+0000","last_became_active":"2024-04-25T15:19:46.485430+0000","last_became_peered":"2024-04-25T15:19:46.485430+0000","last_unstale":"2024-04-25T15:19:52.489275+0000","last_undegraded":"2024-04-25T15:19:52.489275+0000","last_fullsized":"2024-04-25T15:19:52.489275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490843+0000","last_change":"2024-04-25T15:19:45.461658+0000","last_active":"2024-04-25T15:19:52.490843+0000","last_peered":"2024-04-25T15:19:52.490843+0000","last_clean":"2024-04-25T15:19:52.490843+0000","last_became_active":"2024-04-25T15:19:45.461346+0000","last_became_peered":"2024-04-25T15:19:45.461346+0000","last_unstale":"2024-04-25T15:19:52.490843+0000","last_undegraded":"2024-04-25T15:19:52.490843+0000","last_fullsized":"2024-04-25T15:19:52.490843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021048+0000","last_change":"2024-04-25T15:19:50.518359+0000","last_active":"2024-04-25T15:19:54.021048+0000","last_peered":"2024-04-25T15:19:54.021048+0000","last_clean":"2024-04-25T15:19:54.021048+0000","last_became_active":"2024-04-25T15:19:50.517856+0000","last_became_peered":"2024-04-25T15:19:50.517856+0000","last_unstale":"2024-04-25T15:19:54.021048+0000","last_undegraded":"2024-04-25T15:19:54.021048+0000","last_fullsized":"2024-04-25T15:19:54.021048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:08:18.271358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490731+0000","last_change":"2024-04-25T15:19:46.469634+0000","last_active":"2024-04-25T15:19:52.490731+0000","last_peered":"2024-04-25T15:19:52.490731+0000","last_clean":"2024-04-25T15:19:52.490731+0000","last_became_active":"2024-04-25T15:19:46.468979+0000","last_became_peered":"2024-04-25T15:19:46.468979+0000","last_unstale":"2024-04-25T15:19:52.490731+0000","last_undegraded":"2024-04-25T15:19:52.490731+0000","last_fullsized":"2024-04-25T15:19:52.490731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017779+0000","last_change":"2024-04-25T15:19:45.471612+0000","last_active":"2024-04-25T15:19:54.017779+0000","last_peered":"2024-04-25T15:19:54.017779+0000","last_clean":"2024-04-25T15:19:54.017779+0000","last_became_active":"2024-04-25T15:19:45.471289+0000","last_became_peered":"2024-04-25T15:19:45.471289+0000","last_unstale":"2024-04-25T15:19:54.017779+0000","last_undegraded":"2024-04-25T15:19:54.017779+0000","last_fullsized":"2024-04-25T15:19:54.017779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020544+0000","last_change":"2024-04-25T15:19:51.105778+0000","last_active":"2024-04-25T15:19:54.020544+0000","last_peered":"2024-04-25T15:19:54.020544+0000","last_clean":"2024-04-25T15:19:54.020544+0000","last_became_active":"2024-04-25T15:19:51.105435+0000","last_became_peered":"2024-04-25T15:19:51.105435+0000","last_unstale":"2024-04-25T15:19:54.020544+0000","last_undegraded":"2024-04-25T15:19:54.020544+0000","last_fullsized":"2024-04-25T15:19:54.020544+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:09:15.216797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489181+0000","last_change":"2024-04-25T15:19:51.101540+0000","last_active":"2024-04-25T15:19:52.489181+0000","last_peered":"2024-04-25T15:19:52.489181+0000","last_clean":"2024-04-25T15:19:52.489181+0000","last_became_active":"2024-04-25T15:19:51.101126+0000","last_became_peered":"2024-04-25T15:19:51.101126+0000","last_unstale":"2024-04-25T15:19:52.489181+0000","last_undegraded":"2024-04-25T15:19:52.489181+0000","last_fullsized":"2024-04-25T15:19:52.489181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809353+0000","last_change":"2024-04-25T15:19:46.470144+0000","last_active":"2024-04-25T15:19:53.809353+0000","last_peered":"2024-04-25T15:19:53.809353+0000","last_clean":"2024-04-25T15:19:53.809353+0000","last_became_active":"2024-04-25T15:19:46.469786+0000","last_became_peered":"2024-04-25T15:19:46.469786+0000","last_unstale":"2024-04-25T15:19:53.809353+0000","last_undegraded":"2024-04-25T15:19:53.809353+0000","last_fullsized":"2024-04-25T15:19:53.809353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490304+0000","last_change":"2024-04-25T15:19:45.466078+0000","last_active":"2024-04-25T15:19:52.490304+0000","last_peered":"2024-04-25T15:19:52.490304+0000","last_clean":"2024-04-25T15:19:52.490304+0000","last_became_active":"2024-04-25T15:19:45.465744+0000","last_became_peered":"2024-04-25T15:19:45.465744+0000","last_unstale":"2024-04-25T15:19:52.490304+0000","last_undegraded":"2024-04-25T15:19:52.490304+0000","last_fullsized":"2024-04-25T15:19:52.490304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018744+0000","last_change":"2024-04-25T15:19:50.517045+0000","last_active":"2024-04-25T15:19:54.018744+0000","last_peered":"2024-04-25T15:19:54.018744+0000","last_clean":"2024-04-25T15:19:54.018744+0000","last_became_active":"2024-04-25T15:19:50.516571+0000","last_became_peered":"2024-04-25T15:19:50.516571+0000","last_unstale":"2024-04-25T15:19:54.018744+0000","last_undegraded":"2024-04-25T15:19:54.018744+0000","last_fullsized":"2024-04-25T15:19:54.018744+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020866+0000","last_change":"2024-04-25T15:19:46.472281+0000","last_active":"2024-04-25T15:19:54.020866+0000","last_peered":"2024-04-25T15:19:54.020866+0000","last_clean":"2024-04-25T15:19:54.020866+0000","last_became_active":"2024-04-25T15:19:46.471957+0000","last_became_peered":"2024-04-25T15:19:46.471957+0000","last_unstale":"2024-04-25T15:19:54.020866+0000","last_undegraded":"2024-04-25T15:19:54.020866+0000","last_fullsized":"2024-04-25T15:19:54.020866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021072+0000","last_change":"2024-04-25T15:19:45.461678+0000","last_active":"2024-04-25T15:19:54.021072+0000","last_peered":"2024-04-25T15:19:54.021072+0000","last_clean":"2024-04-25T15:19:54.021072+0000","last_became_active":"2024-04-25T15:19:45.461365+0000","last_became_peered":"2024-04-25T15:19:45.461365+0000","last_unstale":"2024-04-25T15:19:54.021072+0000","last_undegraded":"2024-04-25T15:19:54.021072+0000","last_fullsized":"2024-04-25T15:19:54.021072+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.468983+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.468673+0000","last_became_peered":"2024-04-25T15:19:45.468673+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489044+0000","last_change":"2024-04-25T15:19:46.459915+0000","last_active":"2024-04-25T15:19:52.489044+0000","last_peered":"2024-04-25T15:19:52.489044+0000","last_clean":"2024-04-25T15:19:52.489044+0000","last_became_active":"2024-04-25T15:19:46.459596+0000","last_became_peered":"2024-04-25T15:19:46.459596+0000","last_unstale":"2024-04-25T15:19:52.489044+0000","last_undegraded":"2024-04-25T15:19:52.489044+0000","last_fullsized":"2024-04-25T15:19:52.489044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020992+0000","last_change":"2024-04-25T15:19:50.517033+0000","last_active":"2024-04-25T15:19:54.020992+0000","last_peered":"2024-04-25T15:19:54.020992+0000","last_clean":"2024-04-25T15:19:54.020992+0000","last_became_active":"2024-04-25T15:19:50.516587+0000","last_became_peered":"2024-04-25T15:19:50.516587+0000","last_unstale":"2024-04-25T15:19:54.020992+0000","last_undegraded":"2024-04-25T15:19:54.020992+0000","last_fullsized":"2024-04-25T15:19:54.020992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:50.341053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019861+0000","last_change":"2024-04-25T15:19:45.466373+0000","last_active":"2024-04-25T15:19:54.019861+0000","last_peered":"2024-04-25T15:19:54.019861+0000","last_clean":"2024-04-25T15:19:54.019861+0000","last_became_active":"2024-04-25T15:19:45.466054+0000","last_became_peered":"2024-04-25T15:19:45.466054+0000","last_unstale":"2024-04-25T15:19:54.019861+0000","last_undegraded":"2024-04-25T15:19:54.019861+0000","last_fullsized":"2024-04-25T15:19:54.019861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:33.631245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491098+0000","last_change":"2024-04-25T15:19:46.471011+0000","last_active":"2024-04-25T15:19:52.491098+0000","last_peered":"2024-04-25T15:19:52.491098+0000","last_clean":"2024-04-25T15:19:52.491098+0000","last_became_active":"2024-04-25T15:19:46.470620+0000","last_became_peered":"2024-04-25T15:19:46.470620+0000","last_unstale":"2024-04-25T15:19:52.491098+0000","last_undegraded":"2024-04-25T15:19:52.491098+0000","last_fullsized":"2024-04-25T15:19:52.491098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:15.434932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021206+0000","last_change":"2024-04-25T15:19:50.517337+0000","last_active":"2024-04-25T15:19:54.021206+0000","last_peered":"2024-04-25T15:19:54.021206+0000","last_clean":"2024-04-25T15:19:54.021206+0000","last_became_active":"2024-04-25T15:19:50.516848+0000","last_became_peered":"2024-04-25T15:19:50.516848+0000","last_unstale":"2024-04-25T15:19:54.021206+0000","last_undegraded":"2024-04-25T15:19:54.021206+0000","last_fullsized":"2024-04-25T15:19:54.021206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:21:22.692161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018022+0000","last_change":"2024-04-25T15:19:46.476012+0000","last_active":"2024-04-25T15:19:54.018022+0000","last_peered":"2024-04-25T15:19:54.018022+0000","last_clean":"2024-04-25T15:19:54.018022+0000","last_became_active":"2024-04-25T15:19:46.475700+0000","last_became_peered":"2024-04-25T15:19:46.475700+0000","last_unstale":"2024-04-25T15:19:54.018022+0000","last_undegraded":"2024-04-25T15:19:54.018022+0000","last_fullsized":"2024-04-25T15:19:54.018022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020188+0000","last_change":"2024-04-25T15:19:46.484292+0000","last_active":"2024-04-25T15:19:54.020188+0000","last_peered":"2024-04-25T15:19:54.020188+0000","last_clean":"2024-04-25T15:19:54.020188+0000","last_became_active":"2024-04-25T15:19:46.483963+0000","last_became_peered":"2024-04-25T15:19:46.483963+0000","last_unstale":"2024-04-25T15:19:54.020188+0000","last_undegraded":"2024-04-25T15:19:54.020188+0000","last_fullsized":"2024-04-25T15:19:54.020188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490912+0000","last_change":"2024-04-25T15:19:46.484902+0000","last_active":"2024-04-25T15:19:52.490912+0000","last_peered":"2024-04-25T15:19:52.490912+0000","last_clean":"2024-04-25T15:19:52.490912+0000","last_became_active":"2024-04-25T15:19:46.482984+0000","last_became_peered":"2024-04-25T15:19:46.482984+0000","last_unstale":"2024-04-25T15:19:52.490912+0000","last_undegraded":"2024-04-25T15:19:52.490912+0000","last_fullsized":"2024-04-25T15:19:52.490912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488966+0000","last_change":"2024-04-25T15:19:46.468583+0000","last_active":"2024-04-25T15:19:52.488966+0000","last_peered":"2024-04-25T15:19:52.488966+0000","last_clean":"2024-04-25T15:19:52.488966+0000","last_became_active":"2024-04-25T15:19:46.468260+0000","last_became_peered":"2024-04-25T15:19:46.468260+0000","last_unstale":"2024-04-25T15:19:52.488966+0000","last_undegraded":"2024-04-25T15:19:52.488966+0000","last_fullsized":"2024-04-25T15:19:52.488966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021455+0000","last_change":"2024-04-25T15:19:46.478008+0000","last_active":"2024-04-25T15:19:54.021455+0000","last_peered":"2024-04-25T15:19:54.021455+0000","last_clean":"2024-04-25T15:19:54.021455+0000","last_became_active":"2024-04-25T15:19:46.477687+0000","last_became_peered":"2024-04-25T15:19:46.477687+0000","last_unstale":"2024-04-25T15:19:54.021455+0000","last_undegraded":"2024-04-25T15:19:54.021455+0000","last_fullsized":"2024-04-25T15:19:54.021455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809819+0000","last_change":"2024-04-25T15:19:46.475994+0000","last_active":"2024-04-25T15:19:53.809819+0000","last_peered":"2024-04-25T15:19:53.809819+0000","last_clean":"2024-04-25T15:19:53.809819+0000","last_became_active":"2024-04-25T15:19:46.475566+0000","last_became_peered":"2024-04-25T15:19:46.475566+0000","last_unstale":"2024-04-25T15:19:53.809819+0000","last_undegraded":"2024-04-25T15:19:53.809819+0000","last_fullsized":"2024-04-25T15:19:53.809819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018933+0000","last_change":"2024-04-25T15:19:46.480931+0000","last_active":"2024-04-25T15:19:54.018933+0000","last_peered":"2024-04-25T15:19:54.018933+0000","last_clean":"2024-04-25T15:19:54.018933+0000","last_became_active":"2024-04-25T15:19:46.480607+0000","last_became_peered":"2024-04-25T15:19:46.480607+0000","last_unstale":"2024-04-25T15:19:54.018933+0000","last_undegraded":"2024-04-25T15:19:54.018933+0000","last_fullsized":"2024-04-25T15:19:54.018933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020473+0000","last_change":"2024-04-25T15:19:46.478746+0000","last_active":"2024-04-25T15:19:54.020473+0000","last_peered":"2024-04-25T15:19:54.020473+0000","last_clean":"2024-04-25T15:19:54.020473+0000","last_became_active":"2024-04-25T15:19:46.478389+0000","last_became_peered":"2024-04-25T15:19:46.478389+0000","last_unstale":"2024-04-25T15:19:54.020473+0000","last_undegraded":"2024-04-25T15:19:54.020473+0000","last_fullsized":"2024-04-25T15:19:54.020473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490185+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.490185+0000","last_peered":"2024-04-25T15:19:52.490185+0000","last_clean":"2024-04-25T15:19:52.490185+0000","last_became_active":"2024-04-25T15:19:46.482115+0000","last_became_peered":"2024-04-25T15:19:46.482115+0000","last_unstale":"2024-04-25T15:19:52.490185+0000","last_undegraded":"2024-04-25T15:19:52.490185+0000","last_fullsized":"2024-04-25T15:19:52.490185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020755+0000","last_change":"2024-04-25T15:19:46.477554+0000","last_active":"2024-04-25T15:19:54.020755+0000","last_peered":"2024-04-25T15:19:54.020755+0000","last_clean":"2024-04-25T15:19:54.020755+0000","last_became_active":"2024-04-25T15:19:46.477240+0000","last_became_peered":"2024-04-25T15:19:46.477240+0000","last_unstale":"2024-04-25T15:19:54.020755+0000","last_undegraded":"2024-04-25T15:19:54.020755+0000","last_fullsized":"2024-04-25T15:19:54.020755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490811+0000","last_change":"2024-04-25T15:19:46.486093+0000","last_active":"2024-04-25T15:19:52.490811+0000","last_peered":"2024-04-25T15:19:52.490811+0000","last_clean":"2024-04-25T15:19:52.490811+0000","last_became_active":"2024-04-25T15:19:46.484171+0000","last_became_peered":"2024-04-25T15:19:46.484171+0000","last_unstale":"2024-04-25T15:19:52.490811+0000","last_undegraded":"2024-04-25T15:19:52.490811+0000","last_fullsized":"2024-04-25T15:19:52.490811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488875+0000","last_change":"2024-04-25T15:19:46.472207+0000","last_active":"2024-04-25T15:19:52.488875+0000","last_peered":"2024-04-25T15:19:52.488875+0000","last_clean":"2024-04-25T15:19:52.488875+0000","last_became_active":"2024-04-25T15:19:46.471780+0000","last_became_peered":"2024-04-25T15:19:46.471780+0000","last_unstale":"2024-04-25T15:19:52.488875+0000","last_undegraded":"2024-04-25T15:19:52.488875+0000","last_fullsized":"2024-04-25T15:19:52.488875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018502+0000","last_change":"2024-04-25T15:19:46.477605+0000","last_active":"2024-04-25T15:19:54.018502+0000","last_peered":"2024-04-25T15:19:54.018502+0000","last_clean":"2024-04-25T15:19:54.018502+0000","last_became_active":"2024-04-25T15:19:46.477324+0000","last_became_peered":"2024-04-25T15:19:46.477324+0000","last_unstale":"2024-04-25T15:19:54.018502+0000","last_undegraded":"2024-04-25T15:19:54.018502+0000","last_fullsized":"2024-04-25T15:19:54.018502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397007+0000","last_change":"2024-04-25T15:19:46.468508+0000","last_active":"2024-04-25T15:19:53.397007+0000","last_peered":"2024-04-25T15:19:53.397007+0000","last_clean":"2024-04-25T15:19:53.397007+0000","last_became_active":"2024-04-25T15:19:46.468170+0000","last_became_peered":"2024-04-25T15:19:46.468170+0000","last_unstale":"2024-04-25T15:19:53.397007+0000","last_undegraded":"2024-04-25T15:19:53.397007+0000","last_fullsized":"2024-04-25T15:19:53.397007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:00.962714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020658+0000","last_change":"2024-04-25T15:19:46.475011+0000","last_active":"2024-04-25T15:19:54.020658+0000","last_peered":"2024-04-25T15:19:54.020658+0000","last_clean":"2024-04-25T15:19:54.020658+0000","last_became_active":"2024-04-25T15:19:46.474637+0000","last_became_peered":"2024-04-25T15:19:46.474637+0000","last_unstale":"2024-04-25T15:19:54.020658+0000","last_undegraded":"2024-04-25T15:19:54.020658+0000","last_fullsized":"2024-04-25T15:19:54.020658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491965+0000","last_change":"2024-04-25T15:19:46.483654+0000","last_active":"2024-04-25T15:19:52.491965+0000","last_peered":"2024-04-25T15:19:52.491965+0000","last_clean":"2024-04-25T15:19:52.491965+0000","last_became_active":"2024-04-25T15:19:46.483291+0000","last_became_peered":"2024-04-25T15:19:46.483291+0000","last_unstale":"2024-04-25T15:19:52.491965+0000","last_undegraded":"2024-04-25T15:19:52.491965+0000","last_fullsized":"2024-04-25T15:19:52.491965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490226+0000","last_change":"2024-04-25T15:19:46.483513+0000","last_active":"2024-04-25T15:19:52.490226+0000","last_peered":"2024-04-25T15:19:52.490226+0000","last_clean":"2024-04-25T15:19:52.490226+0000","last_became_active":"2024-04-25T15:19:46.483158+0000","last_became_peered":"2024-04-25T15:19:46.483158+0000","last_unstale":"2024-04-25T15:19:52.490226+0000","last_undegraded":"2024-04-25T15:19:52.490226+0000","last_fullsized":"2024-04-25T15:19:52.490226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809438+0000","last_change":"2024-04-25T15:19:46.479148+0000","last_active":"2024-04-25T15:19:53.809438+0000","last_peered":"2024-04-25T15:19:53.809438+0000","last_clean":"2024-04-25T15:19:53.809438+0000","last_became_active":"2024-04-25T15:19:46.478829+0000","last_became_peered":"2024-04-25T15:19:46.478829+0000","last_unstale":"2024-04-25T15:19:53.809438+0000","last_undegraded":"2024-04-25T15:19:53.809438+0000","last_fullsized":"2024-04-25T15:19:53.809438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489010+0000","last_change":"2024-04-25T15:19:46.471858+0000","last_active":"2024-04-25T15:19:52.489010+0000","last_peered":"2024-04-25T15:19:52.489010+0000","last_clean":"2024-04-25T15:19:52.489010+0000","last_became_active":"2024-04-25T15:19:46.471517+0000","last_became_peered":"2024-04-25T15:19:46.471517+0000","last_unstale":"2024-04-25T15:19:52.489010+0000","last_undegraded":"2024-04-25T15:19:52.489010+0000","last_fullsized":"2024-04-25T15:19:52.489010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:58.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020190+0000","last_change":"2024-04-25T15:19:46.473809+0000","last_active":"2024-04-25T15:19:54.020190+0000","last_peered":"2024-04-25T15:19:54.020190+0000","last_clean":"2024-04-25T15:19:54.020190+0000","last_became_active":"2024-04-25T15:19:46.473479+0000","last_became_peered":"2024-04-25T15:19:46.473479+0000","last_unstale":"2024-04-25T15:19:54.020190+0000","last_undegraded":"2024-04-25T15:19:54.020190+0000","last_fullsized":"2024-04-25T15:19:54.020190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490638+0000","last_change":"2024-04-25T15:19:46.473643+0000","last_active":"2024-04-25T15:19:52.490638+0000","last_peered":"2024-04-25T15:19:52.490638+0000","last_clean":"2024-04-25T15:19:52.490638+0000","last_became_active":"2024-04-25T15:19:46.473190+0000","last_became_peered":"2024-04-25T15:19:46.473190+0000","last_unstale":"2024-04-25T15:19:52.490638+0000","last_undegraded":"2024-04-25T15:19:52.490638+0000","last_fullsized":"2024-04-25T15:19:52.490638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018544+0000","last_change":"2024-04-25T15:19:46.488270+0000","last_active":"2024-04-25T15:19:54.018544+0000","last_peered":"2024-04-25T15:19:54.018544+0000","last_clean":"2024-04-25T15:19:54.018544+0000","last_became_active":"2024-04-25T15:19:46.487935+0000","last_became_peered":"2024-04-25T15:19:46.487935+0000","last_unstale":"2024-04-25T15:19:54.018544+0000","last_undegraded":"2024-04-25T15:19:54.018544+0000","last_fullsized":"2024-04-25T15:19:54.018544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018847+0000","last_change":"2024-04-25T15:19:46.488350+0000","last_active":"2024-04-25T15:19:54.018847+0000","last_peered":"2024-04-25T15:19:54.018847+0000","last_clean":"2024-04-25T15:19:54.018847+0000","last_became_active":"2024-04-25T15:19:46.487997+0000","last_became_peered":"2024-04-25T15:19:46.487997+0000","last_unstale":"2024-04-25T15:19:54.018847+0000","last_undegraded":"2024-04-25T15:19:54.018847+0000","last_fullsized":"2024-04-25T15:19:54.018847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397147+0000","last_change":"2024-04-25T15:19:46.486142+0000","last_active":"2024-04-25T15:19:53.397147+0000","last_peered":"2024-04-25T15:19:53.397147+0000","last_clean":"2024-04-25T15:19:53.397147+0000","last_became_active":"2024-04-25T15:19:46.485774+0000","last_became_peered":"2024-04-25T15:19:46.485774+0000","last_unstale":"2024-04-25T15:19:53.397147+0000","last_undegraded":"2024-04-25T15:19:53.397147+0000","last_fullsized":"2024-04-25T15:19:53.397147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:16.297767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020777+0000","last_change":"2024-04-25T15:19:46.486137+0000","last_active":"2024-04-25T15:19:54.020777+0000","last_peered":"2024-04-25T15:19:54.020777+0000","last_clean":"2024-04-25T15:19:54.020777+0000","last_became_active":"2024-04-25T15:19:46.485814+0000","last_became_peered":"2024-04-25T15:19:46.485814+0000","last_unstale":"2024-04-25T15:19:54.020777+0000","last_undegraded":"2024-04-25T15:19:54.020777+0000","last_fullsized":"2024-04-25T15:19:54.020777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.469644+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469320+0000","last_became_peered":"2024-04-25T15:19:46.469320+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490739+0000","last_change":"2024-04-25T15:19:46.486357+0000","last_active":"2024-04-25T15:19:52.490739+0000","last_peered":"2024-04-25T15:19:52.490739+0000","last_clean":"2024-04-25T15:19:52.490739+0000","last_became_active":"2024-04-25T15:19:46.486016+0000","last_became_peered":"2024-04-25T15:19:46.486016+0000","last_unstale":"2024-04-25T15:19:52.490739+0000","last_undegraded":"2024-04-25T15:19:52.490739+0000","last_fullsized":"2024-04-25T15:19:52.490739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.470290+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469965+0000","last_became_peered":"2024-04-25T15:19:46.469965+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491250+0000","last_change":"2024-04-25T15:19:46.483532+0000","last_active":"2024-04-25T15:19:52.491250+0000","last_peered":"2024-04-25T15:19:52.491250+0000","last_clean":"2024-04-25T15:19:52.491250+0000","last_became_active":"2024-04-25T15:19:46.483242+0000","last_became_peered":"2024-04-25T15:19:46.483242+0000","last_unstale":"2024-04-25T15:19:52.491250+0000","last_undegraded":"2024-04-25T15:19:52.491250+0000","last_fullsized":"2024-04-25T15:19:52.491250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489031+0000","last_change":"2024-04-25T15:19:46.467386+0000","last_active":"2024-04-25T15:19:52.489031+0000","last_peered":"2024-04-25T15:19:52.489031+0000","last_clean":"2024-04-25T15:19:52.489031+0000","last_became_active":"2024-04-25T15:19:46.467034+0000","last_became_peered":"2024-04-25T15:19:46.467034+0000","last_unstale":"2024-04-25T15:19:52.489031+0000","last_undegraded":"2024-04-25T15:19:52.489031+0000","last_fullsized":"2024-04-25T15:19:52.489031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809963+0000","last_change":"2024-04-25T15:19:46.472318+0000","last_active":"2024-04-25T15:19:53.809963+0000","last_peered":"2024-04-25T15:19:53.809963+0000","last_clean":"2024-04-25T15:19:53.809963+0000","last_became_active":"2024-04-25T15:19:46.471982+0000","last_became_peered":"2024-04-25T15:19:46.471982+0000","last_unstale":"2024-04-25T15:19:53.809963+0000","last_undegraded":"2024-04-25T15:19:53.809963+0000","last_fullsized":"2024-04-25T15:19:53.809963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397045+0000","last_change":"2024-04-25T15:19:46.473366+0000","last_active":"2024-04-25T15:19:53.397045+0000","last_peered":"2024-04-25T15:19:53.397045+0000","last_clean":"2024-04-25T15:19:53.397045+0000","last_became_active":"2024-04-25T15:19:46.472969+0000","last_became_peered":"2024-04-25T15:19:46.472969+0000","last_unstale":"2024-04-25T15:19:53.397045+0000","last_undegraded":"2024-04-25T15:19:53.397045+0000","last_fullsized":"2024-04-25T15:19:53.397045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:20:09.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:20:09.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.692+0000 7fe7f3bb2640 1 Processor -- start 2024-04-25T15:20:09.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.692+0000 7fe7f3bb2640 1 -- start start 2024-04-25T15:20:09.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.693+0000 7fe7f3bb2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7ec0d2e20 0x7fe7ec0af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.693+0000 7fe7f3bb2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7ec0b4f20 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.693+0000 7fe7f3bb2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7ec0b3a70 con 0x7fe7ec0cfc60 2024-04-25T15:20:09.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.693+0000 7fe7f3bb2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe7ec0b3d50 con 0x7fe7ec0d2e20 2024-04-25T15:20:09.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f214f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7ec0d2e20 0x7fe7ec0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f214f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7ec0d2e20 0x7fe7ec0af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36824/0 (socket says 172.21.15.146:36824) 2024-04-25T15:20:09.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f214f640 1 -- 172.21.15.146:0/604741258 learned_addr learned my addr 172.21.15.146:0/604741258 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:09.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f194e640 1 --1- 172.21.15.146:0/604741258 >> v1:172.21.15.146:6789/0 conn(0x7fe7ec0b4b10 0x7fe7ec0cf550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33990/0 (socket says 172.21.15.146:33990) 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f214f640 1 -- 172.21.15.146:0/604741258 >> v1:172.21.15.177:6789/0 conn(0x7fe7ec0cfc60 legacy=0x7fe7ec0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f214f640 1 -- 172.21.15.146:0/604741258 >> v1:172.21.15.146:6789/0 conn(0x7fe7ec0b4b10 legacy=0x7fe7ec0cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f214f640 1 -- 172.21.15.146:0/604741258 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ec0b4030 con 0x7fe7ec0d2e20 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.694+0000 7fe7f094c640 1 -- 172.21.15.146:0/604741258 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 850387711 0 0) 0x7fe7ec0b4f20 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f214f640 1 --2- 172.21.15.146:0/604741258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0d2e20 0x7fe7ec0af490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009e50 tx=0x7fe7e402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead09ac49f86212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f094c640 1 -- 172.21.15.146:0/604741258 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4036070 con 0x7fe7ec0d2e20 2024-04-25T15:20:09.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f094c640 1 -- 172.21.15.146:0/604741258 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e4031600 con 0x7fe7ec0d2e20 2024-04-25T15:20:09.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f094c640 1 -- 172.21.15.146:0/604741258 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40319d0 con 0x7fe7ec0d2e20 2024-04-25T15:20:09.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/604741258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0d2e20 msgr2=0x7fe7ec0af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/604741258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0d2e20 0x7fe7ec0af490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009e50 tx=0x7fe7e402a6d0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/604741258 shutdown_connections 2024-04-25T15:20:09.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/604741258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0d2e20 0x7fe7ec0af490 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.695+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/604741258 >> 172.21.15.146:0/604741258 conn(0x7fe7ec0b73b0 msgr2=0x7fe7ec0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:09.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.696+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/604741258 shutdown_connections 2024-04-25T15:20:09.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.696+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/604741258 wait complete. 2024-04-25T15:20:09.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.697+0000 7fe7f3bb2640 1 Processor -- start 2024-04-25T15:20:09.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.697+0000 7fe7f3bb2640 1 -- start start 2024-04-25T15:20:09.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 0x7fe7ec15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7ec0d2e20 0x7fe7ec0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7ec159d60 0x7fe7ec0b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec0cefc0 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ec0b45f0 con 0x7fe7ec159d60 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe7ec0cf140 con 0x7fe7ec0d2e20 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 0x7fe7ec15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 0x7fe7ec15bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36838/0 (socket says 172.21.15.146:36838) 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 -- 172.21.15.146:0/3306408624 learned_addr learned my addr 172.21.15.146:0/3306408624 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f214f640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7ec159d60 0x7fe7ec0b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f114d640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7ec0d2e20 0x7fe7ec0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 -- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7ec0d2e20 msgr2=0x7fe7ec0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7ec0d2e20 0x7fe7ec0b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 -- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7ec159d60 msgr2=0x7fe7ec0b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7ec159d60 0x7fe7ec0b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 -- 172.21.15.146:0/3306408624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ec0ca470 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f214f640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7ec159d60 0x7fe7ec0b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f114d640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7ec0d2e20 0x7fe7ec0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f194e640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 0x7fe7ec15bb80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c460 tx=0x7fe7e800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8007480 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e8007620 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ec0ca750 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e80059c0 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.699+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ec0cad10 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.700+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7ec06a0e0 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.701+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe7e8005b60 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.701+0000 7fe7e27fc640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe7d00744e0 0x7fe7d00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.701+0000 7fe7f114d640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe7d00744e0 0x7fe7d00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.701+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe7e8095160 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.702+0000 7fe7f114d640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe7d00744e0 0x7fe7d00769f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc006fd0 tx=0x7fe7dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:09.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.706+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7e805f5b0 con 0x7fe7ec0b4b10 2024-04-25T15:20:09.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.876+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe7ec0ca1f0 con 0x7fe7d00744e0 2024-04-25T15:20:09.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.884+0000 7fe7e27fc640 1 -- 172.21.15.146:0/3306408624 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fe7dc009e40 con 0x7fe7d00744e0 2024-04-25T15:20:09.888 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:09.890 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:20:09.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe7d00744e0 msgr2=0x7fe7d00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe7d00744e0 0x7fe7d00769f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc006fd0 tx=0x7fe7dc008040 comp rx=0 tx=0).stop 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 msgr2=0x7fe7ec15bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 0x7fe7ec15bb80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c460 tx=0x7fe7e800c920 comp rx=0 tx=0).stop 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 shutdown_connections 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe7d00744e0 0x7fe7d00769f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7ec159d60 0x7fe7ec0b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7ec0d2e20 0x7fe7ec0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 --2- 172.21.15.146:0/3306408624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7ec0b4b10 0x7fe7ec15bb80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 >> 172.21.15.146:0/3306408624 conn(0x7fe7ec0b73b0 msgr2=0x7fe7ec071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 shutdown_connections 2024-04-25T15:20:09.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:09.888+0000 7fe7f3bb2640 1 -- 172.21.15.146:0/3306408624 wait complete. 2024-04-25T15:20:09.900 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-25T15:20:07.991797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489335+0000","last_change":"2024-04-25T15:19:45.467889+0000","last_active":"2024-04-25T15:19:52.489335+0000","last_peered":"2024-04-25T15:19:52.489335+0000","last_clean":"2024-04-25T15:19:52.489335+0000","last_became_active":"2024-04-25T15:19:45.467565+0000","last_became_peered":"2024-04-25T15:19:45.467565+0000","last_unstale":"2024-04-25T15:19:52.489335+0000","last_undegraded":"2024-04-25T15:19:52.489335+0000","last_fullsized":"2024-04-25T15:19:52.489335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020856+0000","last_change":"2024-04-25T15:19:46.476186+0000","last_active":"2024-04-25T15:19:54.020856+0000","last_peered":"2024-04-25T15:19:54.020856+0000","last_clean":"2024-04-25T15:19:54.020856+0000","last_became_active":"2024-04-25T15:19:46.475865+0000","last_became_peered":"2024-04-25T15:19:46.475865+0000","last_unstale":"2024-04-25T15:19:54.020856+0000","last_undegraded":"2024-04-25T15:19:54.020856+0000","last_fullsized":"2024-04-25T15:19:54.020856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020377+0000","last_change":"2024-04-25T15:19:51.102406+0000","last_active":"2024-04-25T15:19:54.020377+0000","last_peered":"2024-04-25T15:19:54.020377+0000","last_clean":"2024-04-25T15:19:54.020377+0000","last_became_active":"2024-04-25T15:19:51.101849+0000","last_became_peered":"2024-04-25T15:19:51.101849+0000","last_unstale":"2024-04-25T15:19:54.020377+0000","last_undegraded":"2024-04-25T15:19:54.020377+0000","last_fullsized":"2024-04-25T15:19:54.020377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:58.329289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020474+0000","last_change":"2024-04-25T15:19:45.461077+0000","last_active":"2024-04-25T15:19:54.020474+0000","last_peered":"2024-04-25T15:19:54.020474+0000","last_clean":"2024-04-25T15:19:54.020474+0000","last_became_active":"2024-04-25T15:19:45.460662+0000","last_became_peered":"2024-04-25T15:19:45.460662+0000","last_unstale":"2024-04-25T15:19:54.020474+0000","last_undegraded":"2024-04-25T15:19:54.020474+0000","last_fullsized":"2024-04-25T15:19:54.020474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489205+0000","last_change":"2024-04-25T15:19:46.486683+0000","last_active":"2024-04-25T15:19:52.489205+0000","last_peered":"2024-04-25T15:19:52.489205+0000","last_clean":"2024-04-25T15:19:52.489205+0000","last_became_active":"2024-04-25T15:19:46.486203+0000","last_became_peered":"2024-04-25T15:19:46.486203+0000","last_unstale":"2024-04-25T15:19:52.489205+0000","last_undegraded":"2024-04-25T15:19:52.489205+0000","last_fullsized":"2024-04-25T15:19:52.489205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490288+0000","last_change":"2024-04-25T15:19:50.510692+0000","last_active":"2024-04-25T15:19:52.490288+0000","last_peered":"2024-04-25T15:19:52.490288+0000","last_clean":"2024-04-25T15:19:52.490288+0000","last_became_active":"2024-04-25T15:19:50.510156+0000","last_became_peered":"2024-04-25T15:19:50.510156+0000","last_unstale":"2024-04-25T15:19:52.490288+0000","last_undegraded":"2024-04-25T15:19:52.490288+0000","last_fullsized":"2024-04-25T15:19:52.490288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396965+0000","last_change":"2024-04-25T15:19:45.471053+0000","last_active":"2024-04-25T15:19:53.396965+0000","last_peered":"2024-04-25T15:19:53.396965+0000","last_clean":"2024-04-25T15:19:53.396965+0000","last_became_active":"2024-04-25T15:19:45.470704+0000","last_became_peered":"2024-04-25T15:19:45.470704+0000","last_unstale":"2024-04-25T15:19:53.396965+0000","last_undegraded":"2024-04-25T15:19:53.396965+0000","last_fullsized":"2024-04-25T15:19:53.396965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:55.012973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491359+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.491359+0000","last_peered":"2024-04-25T15:19:52.491359+0000","last_clean":"2024-04-25T15:19:52.491359+0000","last_became_active":"2024-04-25T15:19:46.482181+0000","last_became_peered":"2024-04-25T15:19:46.482181+0000","last_unstale":"2024-04-25T15:19:52.491359+0000","last_undegraded":"2024-04-25T15:19:52.491359+0000","last_fullsized":"2024-04-25T15:19:52.491359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492195+0000","last_change":"2024-04-25T15:19:50.515950+0000","last_active":"2024-04-25T15:19:52.492195+0000","last_peered":"2024-04-25T15:19:52.492195+0000","last_clean":"2024-04-25T15:19:52.492195+0000","last_became_active":"2024-04-25T15:19:50.515486+0000","last_became_peered":"2024-04-25T15:19:50.515486+0000","last_unstale":"2024-04-25T15:19:52.492195+0000","last_undegraded":"2024-04-25T15:19:52.492195+0000","last_fullsized":"2024-04-25T15:19:52.492195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396633+0000","last_change":"2024-04-25T15:19:45.472051+0000","last_active":"2024-04-25T15:19:53.396633+0000","last_peered":"2024-04-25T15:19:53.396633+0000","last_clean":"2024-04-25T15:19:53.396633+0000","last_became_active":"2024-04-25T15:19:45.471710+0000","last_became_peered":"2024-04-25T15:19:45.471710+0000","last_unstale":"2024-04-25T15:19:53.396633+0000","last_undegraded":"2024-04-25T15:19:53.396633+0000","last_fullsized":"2024-04-25T15:19:53.396633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491729+0000","last_change":"2024-04-25T15:19:46.483085+0000","last_active":"2024-04-25T15:19:52.491729+0000","last_peered":"2024-04-25T15:19:52.491729+0000","last_clean":"2024-04-25T15:19:52.491729+0000","last_became_active":"2024-04-25T15:19:46.482749+0000","last_became_peered":"2024-04-25T15:19:46.482749+0000","last_unstale":"2024-04-25T15:19:52.491729+0000","last_undegraded":"2024-04-25T15:19:52.491729+0000","last_fullsized":"2024-04-25T15:19:52.491729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490380+0000","last_change":"2024-04-25T15:19:50.508661+0000","last_active":"2024-04-25T15:19:52.490380+0000","last_peered":"2024-04-25T15:19:52.490380+0000","last_clean":"2024-04-25T15:19:52.490380+0000","last_became_active":"2024-04-25T15:19:50.508116+0000","last_became_peered":"2024-04-25T15:19:50.508116+0000","last_unstale":"2024-04-25T15:19:52.490380+0000","last_undegraded":"2024-04-25T15:19:52.490380+0000","last_fullsized":"2024-04-25T15:19:52.490380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018240+0000","last_change":"2024-04-25T15:19:46.477569+0000","last_active":"2024-04-25T15:19:54.018240+0000","last_peered":"2024-04-25T15:19:54.018240+0000","last_clean":"2024-04-25T15:19:54.018240+0000","last_became_active":"2024-04-25T15:19:46.477188+0000","last_became_peered":"2024-04-25T15:19:46.477188+0000","last_unstale":"2024-04-25T15:19:54.018240+0000","last_undegraded":"2024-04-25T15:19:54.018240+0000","last_fullsized":"2024-04-25T15:19:54.018240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396748+0000","last_change":"2024-04-25T15:19:45.469111+0000","last_active":"2024-04-25T15:19:53.396748+0000","last_peered":"2024-04-25T15:19:53.396748+0000","last_clean":"2024-04-25T15:19:53.396748+0000","last_became_active":"2024-04-25T15:19:45.468780+0000","last_became_peered":"2024-04-25T15:19:45.468780+0000","last_unstale":"2024-04-25T15:19:53.396748+0000","last_undegraded":"2024-04-25T15:19:53.396748+0000","last_fullsized":"2024-04-25T15:19:53.396748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491203+0000","last_change":"2024-04-25T15:19:50.513416+0000","last_active":"2024-04-25T15:19:52.491203+0000","last_peered":"2024-04-25T15:19:52.491203+0000","last_clean":"2024-04-25T15:19:52.491203+0000","last_became_active":"2024-04-25T15:19:50.513024+0000","last_became_peered":"2024-04-25T15:19:50.513024+0000","last_unstale":"2024-04-25T15:19:52.491203+0000","last_undegraded":"2024-04-25T15:19:52.491203+0000","last_fullsized":"2024-04-25T15:19:52.491203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:20.334921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020296+0000","last_change":"2024-04-25T15:19:46.472167+0000","last_active":"2024-04-25T15:19:54.020296+0000","last_peered":"2024-04-25T15:19:54.020296+0000","last_clean":"2024-04-25T15:19:54.020296+0000","last_became_active":"2024-04-25T15:19:46.471839+0000","last_became_peered":"2024-04-25T15:19:46.471839+0000","last_unstale":"2024-04-25T15:19:54.020296+0000","last_undegraded":"2024-04-25T15:19:54.020296+0000","last_fullsized":"2024-04-25T15:19:54.020296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809709+0000","last_change":"2024-04-25T15:19:45.467435+0000","last_active":"2024-04-25T15:19:53.809709+0000","last_peered":"2024-04-25T15:19:53.809709+0000","last_clean":"2024-04-25T15:19:53.809709+0000","last_became_active":"2024-04-25T15:19:45.466960+0000","last_became_peered":"2024-04-25T15:19:45.466960+0000","last_unstale":"2024-04-25T15:19:53.809709+0000","last_undegraded":"2024-04-25T15:19:53.809709+0000","last_fullsized":"2024-04-25T15:19:53.809709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491020+0000","last_change":"2024-04-25T15:19:50.515801+0000","last_active":"2024-04-25T15:19:52.491020+0000","last_peered":"2024-04-25T15:19:52.491020+0000","last_clean":"2024-04-25T15:19:52.491020+0000","last_became_active":"2024-04-25T15:19:50.515340+0000","last_became_peered":"2024-04-25T15:19:50.515340+0000","last_unstale":"2024-04-25T15:19:52.491020+0000","last_undegraded":"2024-04-25T15:19:52.491020+0000","last_fullsized":"2024-04-25T15:19:52.491020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:39.634911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020955+0000","last_change":"2024-04-25T15:19:46.470328+0000","last_active":"2024-04-25T15:19:54.020955+0000","last_peered":"2024-04-25T15:19:54.020955+0000","last_clean":"2024-04-25T15:19:54.020955+0000","last_became_active":"2024-04-25T15:19:46.469972+0000","last_became_peered":"2024-04-25T15:19:46.469972+0000","last_unstale":"2024-04-25T15:19:54.020955+0000","last_undegraded":"2024-04-25T15:19:54.020955+0000","last_fullsized":"2024-04-25T15:19:54.020955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.808963+0000","last_change":"2024-04-25T15:19:45.468754+0000","last_active":"2024-04-25T15:19:53.808963+0000","last_peered":"2024-04-25T15:19:53.808963+0000","last_clean":"2024-04-25T15:19:53.808963+0000","last_became_active":"2024-04-25T15:19:45.468374+0000","last_became_peered":"2024-04-25T15:19:45.468374+0000","last_unstale":"2024-04-25T15:19:53.808963+0000","last_undegraded":"2024-04-25T15:19:53.808963+0000","last_fullsized":"2024-04-25T15:19:53.808963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810131+0000","last_change":"2024-04-25T15:19:51.101484+0000","last_active":"2024-04-25T15:19:53.810131+0000","last_peered":"2024-04-25T15:19:53.810131+0000","last_clean":"2024-04-25T15:19:53.810131+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:53.810131+0000","last_undegraded":"2024-04-25T15:19:53.810131+0000","last_fullsized":"2024-04-25T15:19:53.810131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:57.500581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491979+0000","last_change":"2024-04-25T15:19:46.458762+0000","last_active":"2024-04-25T15:19:52.491979+0000","last_peered":"2024-04-25T15:19:52.491979+0000","last_clean":"2024-04-25T15:19:52.491979+0000","last_became_active":"2024-04-25T15:19:46.458438+0000","last_became_peered":"2024-04-25T15:19:46.458438+0000","last_unstale":"2024-04-25T15:19:52.491979+0000","last_undegraded":"2024-04-25T15:19:52.491979+0000","last_fullsized":"2024-04-25T15:19:52.491979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.402061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396261+0000","last_change":"2024-04-25T15:19:45.471672+0000","last_active":"2024-04-25T15:19:53.396261+0000","last_peered":"2024-04-25T15:19:53.396261+0000","last_clean":"2024-04-25T15:19:53.396261+0000","last_became_active":"2024-04-25T15:19:45.471095+0000","last_became_peered":"2024-04-25T15:19:45.471095+0000","last_unstale":"2024-04-25T15:19:53.396261+0000","last_undegraded":"2024-04-25T15:19:53.396261+0000","last_fullsized":"2024-04-25T15:19:53.396261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490630+0000","last_change":"2024-04-25T15:19:50.511923+0000","last_active":"2024-04-25T15:19:52.490630+0000","last_peered":"2024-04-25T15:19:52.490630+0000","last_clean":"2024-04-25T15:19:52.490630+0000","last_became_active":"2024-04-25T15:19:50.511350+0000","last_became_peered":"2024-04-25T15:19:50.511350+0000","last_unstale":"2024-04-25T15:19:52.490630+0000","last_undegraded":"2024-04-25T15:19:52.490630+0000","last_fullsized":"2024-04-25T15:19:52.490630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489343+0000","last_change":"2024-04-25T15:19:46.472562+0000","last_active":"2024-04-25T15:19:52.489343+0000","last_peered":"2024-04-25T15:19:52.489343+0000","last_clean":"2024-04-25T15:19:52.489343+0000","last_became_active":"2024-04-25T15:19:46.471397+0000","last_became_peered":"2024-04-25T15:19:46.471397+0000","last_unstale":"2024-04-25T15:19:52.489343+0000","last_undegraded":"2024-04-25T15:19:52.489343+0000","last_fullsized":"2024-04-25T15:19:52.489343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017764+0000","last_change":"2024-04-25T15:19:45.467859+0000","last_active":"2024-04-25T15:19:54.017764+0000","last_peered":"2024-04-25T15:19:54.017764+0000","last_clean":"2024-04-25T15:19:54.017764+0000","last_became_active":"2024-04-25T15:19:45.467461+0000","last_became_peered":"2024-04-25T15:19:45.467461+0000","last_unstale":"2024-04-25T15:19:54.017764+0000","last_undegraded":"2024-04-25T15:19:54.017764+0000","last_fullsized":"2024-04-25T15:19:54.017764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491639+0000","last_change":"2024-04-25T15:19:51.101973+0000","last_active":"2024-04-25T15:19:52.491639+0000","last_peered":"2024-04-25T15:19:52.491639+0000","last_clean":"2024-04-25T15:19:52.491639+0000","last_became_active":"2024-04-25T15:19:51.101046+0000","last_became_peered":"2024-04-25T15:19:51.101046+0000","last_unstale":"2024-04-25T15:19:52.491639+0000","last_undegraded":"2024-04-25T15:19:52.491639+0000","last_fullsized":"2024-04-25T15:19:52.491639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020585+0000","last_change":"2024-04-25T15:19:46.481579+0000","last_active":"2024-04-25T15:19:54.020585+0000","last_peered":"2024-04-25T15:19:54.020585+0000","last_clean":"2024-04-25T15:19:54.020585+0000","last_became_active":"2024-04-25T15:19:46.481265+0000","last_became_peered":"2024-04-25T15:19:46.481265+0000","last_unstale":"2024-04-25T15:19:54.020585+0000","last_undegraded":"2024-04-25T15:19:54.020585+0000","last_fullsized":"2024-04-25T15:19:54.020585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810014+0000","last_change":"2024-04-25T15:19:45.464607+0000","last_active":"2024-04-25T15:19:53.810014+0000","last_peered":"2024-04-25T15:19:53.810014+0000","last_clean":"2024-04-25T15:19:53.810014+0000","last_became_active":"2024-04-25T15:19:45.464278+0000","last_became_peered":"2024-04-25T15:19:45.464278+0000","last_unstale":"2024-04-25T15:19:53.810014+0000","last_undegraded":"2024-04-25T15:19:53.810014+0000","last_fullsized":"2024-04-25T15:19:53.810014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020107+0000","last_change":"2024-04-25T15:19:51.101501+0000","last_active":"2024-04-25T15:19:54.020107+0000","last_peered":"2024-04-25T15:19:54.020107+0000","last_clean":"2024-04-25T15:19:54.020107+0000","last_became_active":"2024-04-25T15:19:51.101130+0000","last_became_peered":"2024-04-25T15:19:51.101130+0000","last_unstale":"2024-04-25T15:19:54.020107+0000","last_undegraded":"2024-04-25T15:19:54.020107+0000","last_fullsized":"2024-04-25T15:19:54.020107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809775+0000","last_change":"2024-04-25T15:19:46.469600+0000","last_active":"2024-04-25T15:19:53.809775+0000","last_peered":"2024-04-25T15:19:53.809775+0000","last_clean":"2024-04-25T15:19:53.809775+0000","last_became_active":"2024-04-25T15:19:46.469254+0000","last_became_peered":"2024-04-25T15:19:46.469254+0000","last_unstale":"2024-04-25T15:19:53.809775+0000","last_undegraded":"2024-04-25T15:19:53.809775+0000","last_fullsized":"2024-04-25T15:19:53.809775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397107+0000","last_change":"2024-04-25T15:19:45.471728+0000","last_active":"2024-04-25T15:19:53.397107+0000","last_peered":"2024-04-25T15:19:53.397107+0000","last_clean":"2024-04-25T15:19:53.397107+0000","last_became_active":"2024-04-25T15:19:45.471426+0000","last_became_peered":"2024-04-25T15:19:45.471426+0000","last_unstale":"2024-04-25T15:19:53.397107+0000","last_undegraded":"2024-04-25T15:19:53.397107+0000","last_fullsized":"2024-04-25T15:19:53.397107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:39:45.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488880+0000","last_change":"2024-04-25T15:19:51.101520+0000","last_active":"2024-04-25T15:19:52.488880+0000","last_peered":"2024-04-25T15:19:52.488880+0000","last_clean":"2024-04-25T15:19:52.488880+0000","last_became_active":"2024-04-25T15:19:51.100989+0000","last_became_peered":"2024-04-25T15:19:51.100989+0000","last_unstale":"2024-04-25T15:19:52.488880+0000","last_undegraded":"2024-04-25T15:19:52.488880+0000","last_fullsized":"2024-04-25T15:19:52.488880+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491850+0000","last_change":"2024-04-25T15:19:46.482591+0000","last_active":"2024-04-25T15:19:52.491850+0000","last_peered":"2024-04-25T15:19:52.491850+0000","last_clean":"2024-04-25T15:19:52.491850+0000","last_became_active":"2024-04-25T15:19:46.482265+0000","last_became_peered":"2024-04-25T15:19:46.482265+0000","last_unstale":"2024-04-25T15:19:52.491850+0000","last_undegraded":"2024-04-25T15:19:52.491850+0000","last_fullsized":"2024-04-25T15:19:52.491850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809000+0000","last_change":"2024-04-25T15:19:45.470984+0000","last_active":"2024-04-25T15:19:53.809000+0000","last_peered":"2024-04-25T15:19:53.809000+0000","last_clean":"2024-04-25T15:19:53.809000+0000","last_became_active":"2024-04-25T15:19:45.470665+0000","last_became_peered":"2024-04-25T15:19:45.470665+0000","last_unstale":"2024-04-25T15:19:53.809000+0000","last_undegraded":"2024-04-25T15:19:53.809000+0000","last_fullsized":"2024-04-25T15:19:53.809000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809497+0000","last_change":"2024-04-25T15:19:50.509138+0000","last_active":"2024-04-25T15:19:53.809497+0000","last_peered":"2024-04-25T15:19:53.809497+0000","last_clean":"2024-04-25T15:19:53.809497+0000","last_became_active":"2024-04-25T15:19:50.508713+0000","last_became_peered":"2024-04-25T15:19:50.508713+0000","last_unstale":"2024-04-25T15:19:53.809497+0000","last_undegraded":"2024-04-25T15:19:53.809497+0000","last_fullsized":"2024-04-25T15:19:53.809497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492266+0000","last_change":"2024-04-25T15:19:46.459065+0000","last_active":"2024-04-25T15:19:52.492266+0000","last_peered":"2024-04-25T15:19:52.492266+0000","last_clean":"2024-04-25T15:19:52.492266+0000","last_became_active":"2024-04-25T15:19:46.458708+0000","last_became_peered":"2024-04-25T15:19:46.458708+0000","last_unstale":"2024-04-25T15:19:52.492266+0000","last_undegraded":"2024-04-25T15:19:52.492266+0000","last_fullsized":"2024-04-25T15:19:52.492266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:16.345566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491129+0000","last_change":"2024-04-25T15:19:45.464370+0000","last_active":"2024-04-25T15:19:52.491129+0000","last_peered":"2024-04-25T15:19:52.491129+0000","last_clean":"2024-04-25T15:19:52.491129+0000","last_became_active":"2024-04-25T15:19:45.464041+0000","last_became_peered":"2024-04-25T15:19:45.464041+0000","last_unstale":"2024-04-25T15:19:52.491129+0000","last_undegraded":"2024-04-25T15:19:52.491129+0000","last_fullsized":"2024-04-25T15:19:52.491129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809793+0000","last_change":"2024-04-25T15:19:50.516748+0000","last_active":"2024-04-25T15:19:53.809793+0000","last_peered":"2024-04-25T15:19:53.809793+0000","last_clean":"2024-04-25T15:19:53.809793+0000","last_became_active":"2024-04-25T15:19:50.515480+0000","last_became_peered":"2024-04-25T15:19:50.515480+0000","last_unstale":"2024-04-25T15:19:53.809793+0000","last_undegraded":"2024-04-25T15:19:53.809793+0000","last_fullsized":"2024-04-25T15:19:53.809793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:54.507021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809413+0000","last_change":"2024-04-25T15:19:46.469618+0000","last_active":"2024-04-25T15:19:53.809413+0000","last_peered":"2024-04-25T15:19:53.809413+0000","last_clean":"2024-04-25T15:19:53.809413+0000","last_became_active":"2024-04-25T15:19:46.469289+0000","last_became_peered":"2024-04-25T15:19:46.469289+0000","last_unstale":"2024-04-25T15:19:53.809413+0000","last_undegraded":"2024-04-25T15:19:53.809413+0000","last_fullsized":"2024-04-25T15:19:53.809413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020301+0000","last_change":"2024-04-25T15:19:45.462951+0000","last_active":"2024-04-25T15:19:54.020301+0000","last_peered":"2024-04-25T15:19:54.020301+0000","last_clean":"2024-04-25T15:19:54.020301+0000","last_became_active":"2024-04-25T15:19:45.462424+0000","last_became_peered":"2024-04-25T15:19:45.462424+0000","last_unstale":"2024-04-25T15:19:54.020301+0000","last_undegraded":"2024-04-25T15:19:54.020301+0000","last_fullsized":"2024-04-25T15:19:54.020301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810177+0000","last_change":"2024-04-25T15:19:50.519794+0000","last_active":"2024-04-25T15:19:53.810177+0000","last_peered":"2024-04-25T15:19:53.810177+0000","last_clean":"2024-04-25T15:19:53.810177+0000","last_became_active":"2024-04-25T15:19:50.519278+0000","last_became_peered":"2024-04-25T15:19:50.519278+0000","last_unstale":"2024-04-25T15:19:53.810177+0000","last_undegraded":"2024-04-25T15:19:53.810177+0000","last_fullsized":"2024-04-25T15:19:53.810177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488686+0000","last_change":"2024-04-25T15:19:46.459839+0000","last_active":"2024-04-25T15:19:52.488686+0000","last_peered":"2024-04-25T15:19:52.488686+0000","last_clean":"2024-04-25T15:19:52.488686+0000","last_became_active":"2024-04-25T15:19:46.459401+0000","last_became_peered":"2024-04-25T15:19:46.459401+0000","last_unstale":"2024-04-25T15:19:52.488686+0000","last_undegraded":"2024-04-25T15:19:52.488686+0000","last_fullsized":"2024-04-25T15:19:52.488686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:46.303899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396291+0000","last_change":"2024-04-25T15:19:45.471412+0000","last_active":"2024-04-25T15:19:53.396291+0000","last_peered":"2024-04-25T15:19:53.396291+0000","last_clean":"2024-04-25T15:19:53.396291+0000","last_became_active":"2024-04-25T15:19:45.470632+0000","last_became_peered":"2024-04-25T15:19:45.470632+0000","last_unstale":"2024-04-25T15:19:53.396291+0000","last_undegraded":"2024-04-25T15:19:53.396291+0000","last_fullsized":"2024-04-25T15:19:53.396291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489312+0000","last_change":"2024-04-25T15:19:51.101653+0000","last_active":"2024-04-25T15:19:52.489312+0000","last_peered":"2024-04-25T15:19:52.489312+0000","last_clean":"2024-04-25T15:19:52.489312+0000","last_became_active":"2024-04-25T15:19:51.101246+0000","last_became_peered":"2024-04-25T15:19:51.101246+0000","last_unstale":"2024-04-25T15:19:52.489312+0000","last_undegraded":"2024-04-25T15:19:52.489312+0000","last_fullsized":"2024-04-25T15:19:52.489312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020754+0000","last_change":"2024-04-25T15:19:46.488866+0000","last_active":"2024-04-25T15:19:54.020754+0000","last_peered":"2024-04-25T15:19:54.020754+0000","last_clean":"2024-04-25T15:19:54.020754+0000","last_became_active":"2024-04-25T15:19:46.488494+0000","last_became_peered":"2024-04-25T15:19:46.488494+0000","last_unstale":"2024-04-25T15:19:54.020754+0000","last_undegraded":"2024-04-25T15:19:54.020754+0000","last_fullsized":"2024-04-25T15:19:54.020754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490534+0000","last_change":"2024-04-25T15:19:45.459379+0000","last_active":"2024-04-25T15:19:52.490534+0000","last_peered":"2024-04-25T15:19:52.490534+0000","last_clean":"2024-04-25T15:19:52.490534+0000","last_became_active":"2024-04-25T15:19:45.458985+0000","last_became_peered":"2024-04-25T15:19:45.458985+0000","last_unstale":"2024-04-25T15:19:52.490534+0000","last_undegraded":"2024-04-25T15:19:52.490534+0000","last_fullsized":"2024-04-25T15:19:52.490534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396508+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:19:53.396508+0000","last_peered":"2024-04-25T15:19:53.396508+0000","last_clean":"2024-04-25T15:19:53.396508+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:19:53.396508+0000","last_undegraded":"2024-04-25T15:19:53.396508+0000","last_fullsized":"2024-04-25T15:19:53.396508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018620+0000","last_change":"2024-04-25T15:19:51.102362+0000","last_active":"2024-04-25T15:19:54.018620+0000","last_peered":"2024-04-25T15:19:54.018620+0000","last_clean":"2024-04-25T15:19:54.018620+0000","last_became_active":"2024-04-25T15:19:51.101137+0000","last_became_peered":"2024-04-25T15:19:51.101137+0000","last_unstale":"2024-04-25T15:19:54.018620+0000","last_undegraded":"2024-04-25T15:19:54.018620+0000","last_fullsized":"2024-04-25T15:19:54.018620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396808+0000","last_change":"2024-04-25T15:19:46.486533+0000","last_active":"2024-04-25T15:19:53.396808+0000","last_peered":"2024-04-25T15:19:53.396808+0000","last_clean":"2024-04-25T15:19:53.396808+0000","last_became_active":"2024-04-25T15:19:46.485775+0000","last_became_peered":"2024-04-25T15:19:46.485775+0000","last_unstale":"2024-04-25T15:19:53.396808+0000","last_undegraded":"2024-04-25T15:19:53.396808+0000","last_fullsized":"2024-04-25T15:19:53.396808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019912+0000","last_change":"2024-04-25T15:19:45.467231+0000","last_active":"2024-04-25T15:19:54.019912+0000","last_peered":"2024-04-25T15:19:54.019912+0000","last_clean":"2024-04-25T15:19:54.019912+0000","last_became_active":"2024-04-25T15:19:45.466874+0000","last_became_peered":"2024-04-25T15:19:45.466874+0000","last_unstale":"2024-04-25T15:19:54.019912+0000","last_undegraded":"2024-04-25T15:19:54.019912+0000","last_fullsized":"2024-04-25T15:19:54.019912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018302+0000","last_change":"2024-04-25T15:19:51.101630+0000","last_active":"2024-04-25T15:19:54.018302+0000","last_peered":"2024-04-25T15:19:54.018302+0000","last_clean":"2024-04-25T15:19:54.018302+0000","last_became_active":"2024-04-25T15:19:51.101121+0000","last_became_peered":"2024-04-25T15:19:51.101121+0000","last_unstale":"2024-04-25T15:19:54.018302+0000","last_undegraded":"2024-04-25T15:19:54.018302+0000","last_fullsized":"2024-04-25T15:19:54.018302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491492+0000","last_change":"2024-04-25T15:19:46.483183+0000","last_active":"2024-04-25T15:19:52.491492+0000","last_peered":"2024-04-25T15:19:52.491492+0000","last_clean":"2024-04-25T15:19:52.491492+0000","last_became_active":"2024-04-25T15:19:46.482618+0000","last_became_peered":"2024-04-25T15:19:46.482618+0000","last_unstale":"2024-04-25T15:19:52.491492+0000","last_undegraded":"2024-04-25T15:19:52.491492+0000","last_fullsized":"2024-04-25T15:19:52.491492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809024+0000","last_change":"2024-04-25T15:19:45.469018+0000","last_active":"2024-04-25T15:19:53.809024+0000","last_peered":"2024-04-25T15:19:53.809024+0000","last_clean":"2024-04-25T15:19:53.809024+0000","last_became_active":"2024-04-25T15:19:45.468712+0000","last_became_peered":"2024-04-25T15:19:45.468712+0000","last_unstale":"2024-04-25T15:19:53.809024+0000","last_undegraded":"2024-04-25T15:19:53.809024+0000","last_fullsized":"2024-04-25T15:19:53.809024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489037+0000","last_change":"2024-04-25T15:19:50.513573+0000","last_active":"2024-04-25T15:19:52.489037+0000","last_peered":"2024-04-25T15:19:52.489037+0000","last_clean":"2024-04-25T15:19:52.489037+0000","last_became_active":"2024-04-25T15:19:50.513018+0000","last_became_peered":"2024-04-25T15:19:50.513018+0000","last_unstale":"2024-04-25T15:19:52.489037+0000","last_undegraded":"2024-04-25T15:19:52.489037+0000","last_fullsized":"2024-04-25T15:19:52.489037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397172+0000","last_change":"2024-04-25T15:19:46.465000+0000","last_active":"2024-04-25T15:19:53.397172+0000","last_peered":"2024-04-25T15:19:53.397172+0000","last_clean":"2024-04-25T15:19:53.397172+0000","last_became_active":"2024-04-25T15:19:46.464680+0000","last_became_peered":"2024-04-25T15:19:46.464680+0000","last_unstale":"2024-04-25T15:19:53.397172+0000","last_undegraded":"2024-04-25T15:19:53.397172+0000","last_fullsized":"2024-04-25T15:19:53.397172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:35.597757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020764+0000","last_change":"2024-04-25T15:19:45.463067+0000","last_active":"2024-04-25T15:19:54.020764+0000","last_peered":"2024-04-25T15:19:54.020764+0000","last_clean":"2024-04-25T15:19:54.020764+0000","last_became_active":"2024-04-25T15:19:45.462611+0000","last_became_peered":"2024-04-25T15:19:45.462611+0000","last_unstale":"2024-04-25T15:19:54.020764+0000","last_undegraded":"2024-04-25T15:19:54.020764+0000","last_fullsized":"2024-04-25T15:19:54.020764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490099+0000","last_change":"2024-04-25T15:19:51.101470+0000","last_active":"2024-04-25T15:19:52.490099+0000","last_peered":"2024-04-25T15:19:52.490099+0000","last_clean":"2024-04-25T15:19:52.490099+0000","last_became_active":"2024-04-25T15:19:51.100991+0000","last_became_peered":"2024-04-25T15:19:51.100991+0000","last_unstale":"2024-04-25T15:19:52.490099+0000","last_undegraded":"2024-04-25T15:19:52.490099+0000","last_fullsized":"2024-04-25T15:19:52.490099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396886+0000","last_change":"2024-04-25T15:19:46.478228+0000","last_active":"2024-04-25T15:19:53.396886+0000","last_peered":"2024-04-25T15:19:53.396886+0000","last_clean":"2024-04-25T15:19:53.396886+0000","last_became_active":"2024-04-25T15:19:46.477939+0000","last_became_peered":"2024-04-25T15:19:46.477939+0000","last_unstale":"2024-04-25T15:19:53.396886+0000","last_undegraded":"2024-04-25T15:19:53.396886+0000","last_fullsized":"2024-04-25T15:19:53.396886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491269+0000","last_change":"2024-04-25T15:19:45.456572+0000","last_active":"2024-04-25T15:19:52.491269+0000","last_peered":"2024-04-25T15:19:52.491269+0000","last_clean":"2024-04-25T15:19:52.491269+0000","last_became_active":"2024-04-25T15:19:45.456238+0000","last_became_peered":"2024-04-25T15:19:45.456238+0000","last_unstale":"2024-04-25T15:19:52.491269+0000","last_undegraded":"2024-04-25T15:19:52.491269+0000","last_fullsized":"2024-04-25T15:19:52.491269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:09.326979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491720+0000","last_change":"2024-04-25T15:19:50.511879+0000","last_active":"2024-04-25T15:19:52.491720+0000","last_peered":"2024-04-25T15:19:52.491720+0000","last_clean":"2024-04-25T15:19:52.491720+0000","last_became_active":"2024-04-25T15:19:50.511380+0000","last_became_peered":"2024-04-25T15:19:50.511380+0000","last_unstale":"2024-04-25T15:19:52.491720+0000","last_undegraded":"2024-04-25T15:19:52.491720+0000","last_fullsized":"2024-04-25T15:19:52.491720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396446+0000","last_change":"2024-04-25T15:19:46.465026+0000","last_active":"2024-04-25T15:19:53.396446+0000","last_peered":"2024-04-25T15:19:53.396446+0000","last_clean":"2024-04-25T15:19:53.396446+0000","last_became_active":"2024-04-25T15:19:46.464703+0000","last_became_peered":"2024-04-25T15:19:46.464703+0000","last_unstale":"2024-04-25T15:19:53.396446+0000","last_undegraded":"2024-04-25T15:19:53.396446+0000","last_fullsized":"2024-04-25T15:19:53.396446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020165+0000","last_change":"2024-04-25T15:19:45.465017+0000","last_active":"2024-04-25T15:19:54.020165+0000","last_peered":"2024-04-25T15:19:54.020165+0000","last_clean":"2024-04-25T15:19:54.020165+0000","last_became_active":"2024-04-25T15:19:45.464691+0000","last_became_peered":"2024-04-25T15:19:45.464691+0000","last_unstale":"2024-04-25T15:19:54.020165+0000","last_undegraded":"2024-04-25T15:19:54.020165+0000","last_fullsized":"2024-04-25T15:19:54.020165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489222+0000","last_change":"2024-04-25T15:19:50.508942+0000","last_active":"2024-04-25T15:19:52.489222+0000","last_peered":"2024-04-25T15:19:52.489222+0000","last_clean":"2024-04-25T15:19:52.489222+0000","last_became_active":"2024-04-25T15:19:50.508482+0000","last_became_peered":"2024-04-25T15:19:50.508482+0000","last_unstale":"2024-04-25T15:19:52.489222+0000","last_undegraded":"2024-04-25T15:19:52.489222+0000","last_fullsized":"2024-04-25T15:19:52.489222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018243+0000","last_change":"2024-04-25T15:19:46.473708+0000","last_active":"2024-04-25T15:19:54.018243+0000","last_peered":"2024-04-25T15:19:54.018243+0000","last_clean":"2024-04-25T15:19:54.018243+0000","last_became_active":"2024-04-25T15:19:46.473372+0000","last_became_peered":"2024-04-25T15:19:46.473372+0000","last_unstale":"2024-04-25T15:19:54.018243+0000","last_undegraded":"2024-04-25T15:19:54.018243+0000","last_fullsized":"2024-04-25T15:19:54.018243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.469745+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.469458+0000","last_became_peered":"2024-04-25T15:19:45.469458+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020364+0000","last_change":"2024-04-25T15:19:51.101455+0000","last_active":"2024-04-25T15:19:54.020364+0000","last_peered":"2024-04-25T15:19:54.020364+0000","last_clean":"2024-04-25T15:19:54.020364+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:54.020364+0000","last_undegraded":"2024-04-25T15:19:54.020364+0000","last_fullsized":"2024-04-25T15:19:54.020364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:05.089655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396638+0000","last_change":"2024-04-25T15:19:46.465773+0000","last_active":"2024-04-25T15:19:53.396638+0000","last_peered":"2024-04-25T15:19:53.396638+0000","last_clean":"2024-04-25T15:19:53.396638+0000","last_became_active":"2024-04-25T15:19:46.465467+0000","last_became_peered":"2024-04-25T15:19:46.465467+0000","last_unstale":"2024-04-25T15:19:53.396638+0000","last_undegraded":"2024-04-25T15:19:53.396638+0000","last_fullsized":"2024-04-25T15:19:53.396638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491007+0000","last_change":"2024-04-25T15:19:45.463534+0000","last_active":"2024-04-25T15:19:52.491007+0000","last_peered":"2024-04-25T15:19:52.491007+0000","last_clean":"2024-04-25T15:19:52.491007+0000","last_became_active":"2024-04-25T15:19:45.463167+0000","last_became_peered":"2024-04-25T15:19:45.463167+0000","last_unstale":"2024-04-25T15:19:52.491007+0000","last_undegraded":"2024-04-25T15:19:52.491007+0000","last_fullsized":"2024-04-25T15:19:52.491007+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021379+0000","last_change":"2024-04-25T15:19:51.102717+0000","last_active":"2024-04-25T15:19:54.021379+0000","last_peered":"2024-04-25T15:19:54.021379+0000","last_clean":"2024-04-25T15:19:54.021379+0000","last_became_active":"2024-04-25T15:19:51.102247+0000","last_became_peered":"2024-04-25T15:19:51.102247+0000","last_unstale":"2024-04-25T15:19:54.021379+0000","last_undegraded":"2024-04-25T15:19:54.021379+0000","last_fullsized":"2024-04-25T15:19:54.021379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:14:00.372742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489038+0000","last_change":"2024-04-25T15:19:46.460776+0000","last_active":"2024-04-25T15:19:52.489038+0000","last_peered":"2024-04-25T15:19:52.489038+0000","last_clean":"2024-04-25T15:19:52.489038+0000","last_became_active":"2024-04-25T15:19:46.460453+0000","last_became_peered":"2024-04-25T15:19:46.460453+0000","last_unstale":"2024-04-25T15:19:52.489038+0000","last_undegraded":"2024-04-25T15:19:52.489038+0000","last_fullsized":"2024-04-25T15:19:52.489038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489102+0000","last_change":"2024-04-25T15:19:45.462749+0000","last_active":"2024-04-25T15:19:52.489102+0000","last_peered":"2024-04-25T15:19:52.489102+0000","last_clean":"2024-04-25T15:19:52.489102+0000","last_became_active":"2024-04-25T15:19:45.462417+0000","last_became_peered":"2024-04-25T15:19:45.462417+0000","last_unstale":"2024-04-25T15:19:52.489102+0000","last_undegraded":"2024-04-25T15:19:52.489102+0000","last_fullsized":"2024-04-25T15:19:52.489102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491884+0000","last_change":"2024-04-25T15:19:51.102630+0000","last_active":"2024-04-25T15:19:52.491884+0000","last_peered":"2024-04-25T15:19:52.491884+0000","last_clean":"2024-04-25T15:19:52.491884+0000","last_became_active":"2024-04-25T15:19:51.101912+0000","last_became_peered":"2024-04-25T15:19:51.101912+0000","last_unstale":"2024-04-25T15:19:52.491884+0000","last_undegraded":"2024-04-25T15:19:52.491884+0000","last_fullsized":"2024-04-25T15:19:52.491884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396712+0000","last_change":"2024-04-25T15:19:46.471583+0000","last_active":"2024-04-25T15:19:53.396712+0000","last_peered":"2024-04-25T15:19:53.396712+0000","last_clean":"2024-04-25T15:19:53.396712+0000","last_became_active":"2024-04-25T15:19:46.471286+0000","last_became_peered":"2024-04-25T15:19:46.471286+0000","last_unstale":"2024-04-25T15:19:53.396712+0000","last_undegraded":"2024-04-25T15:19:53.396712+0000","last_fullsized":"2024-04-25T15:19:53.396712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:55.948605+0000","last_change":"2024-04-25T15:19:45.464680+0000","last_active":"2024-04-25T15:19:55.948605+0000","last_peered":"2024-04-25T15:19:55.948605+0000","last_clean":"2024-04-25T15:19:55.948605+0000","last_became_active":"2024-04-25T15:19:45.463690+0000","last_became_peered":"2024-04-25T15:19:45.463690+0000","last_unstale":"2024-04-25T15:19:55.948605+0000","last_undegraded":"2024-04-25T15:19:55.948605+0000","last_fullsized":"2024-04-25T15:19:55.948605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809637+0000","last_change":"2024-04-25T15:19:50.509907+0000","last_active":"2024-04-25T15:19:53.809637+0000","last_peered":"2024-04-25T15:19:53.809637+0000","last_clean":"2024-04-25T15:19:53.809637+0000","last_became_active":"2024-04-25T15:19:50.508793+0000","last_became_peered":"2024-04-25T15:19:50.508793+0000","last_unstale":"2024-04-25T15:19:53.809637+0000","last_undegraded":"2024-04-25T15:19:53.809637+0000","last_fullsized":"2024-04-25T15:19:53.809637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490746+0000","last_change":"2024-04-25T15:19:46.471815+0000","last_active":"2024-04-25T15:19:52.490746+0000","last_peered":"2024-04-25T15:19:52.490746+0000","last_clean":"2024-04-25T15:19:52.490746+0000","last_became_active":"2024-04-25T15:19:46.471473+0000","last_became_peered":"2024-04-25T15:19:46.471473+0000","last_unstale":"2024-04-25T15:19:52.490746+0000","last_undegraded":"2024-04-25T15:19:52.490746+0000","last_fullsized":"2024-04-25T15:19:52.490746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809045+0000","last_change":"2024-04-25T15:19:45.472821+0000","last_active":"2024-04-25T15:19:53.809045+0000","last_peered":"2024-04-25T15:19:53.809045+0000","last_clean":"2024-04-25T15:19:53.809045+0000","last_became_active":"2024-04-25T15:19:45.472420+0000","last_became_peered":"2024-04-25T15:19:45.472420+0000","last_unstale":"2024-04-25T15:19:53.809045+0000","last_undegraded":"2024-04-25T15:19:53.809045+0000","last_fullsized":"2024-04-25T15:19:53.809045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492136+0000","last_change":"2024-04-25T15:19:50.509990+0000","last_active":"2024-04-25T15:19:52.492136+0000","last_peered":"2024-04-25T15:19:52.492136+0000","last_clean":"2024-04-25T15:19:52.492136+0000","last_became_active":"2024-04-25T15:19:50.509340+0000","last_became_peered":"2024-04-25T15:19:50.509340+0000","last_unstale":"2024-04-25T15:19:52.492136+0000","last_undegraded":"2024-04-25T15:19:52.492136+0000","last_fullsized":"2024-04-25T15:19:52.492136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489275+0000","last_change":"2024-04-25T15:19:46.485819+0000","last_active":"2024-04-25T15:19:52.489275+0000","last_peered":"2024-04-25T15:19:52.489275+0000","last_clean":"2024-04-25T15:19:52.489275+0000","last_became_active":"2024-04-25T15:19:46.485430+0000","last_became_peered":"2024-04-25T15:19:46.485430+0000","last_unstale":"2024-04-25T15:19:52.489275+0000","last_undegraded":"2024-04-25T15:19:52.489275+0000","last_fullsized":"2024-04-25T15:19:52.489275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490843+0000","last_change":"2024-04-25T15:19:45.461658+0000","last_active":"2024-04-25T15:19:52.490843+0000","last_peered":"2024-04-25T15:19:52.490843+0000","last_clean":"2024-04-25T15:19:52.490843+0000","last_became_active":"2024-04-25T15:19:45.461346+0000","last_became_peered":"2024-04-25T15:19:45.461346+0000","last_unstale":"2024-04-25T15:19:52.490843+0000","last_undegraded":"2024-04-25T15:19:52.490843+0000","last_fullsized":"2024-04-25T15:19:52.490843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021048+0000","last_change":"2024-04-25T15:19:50.518359+0000","last_active":"2024-04-25T15:19:54.021048+0000","last_peered":"2024-04-25T15:19:54.021048+0000","last_clean":"2024-04-25T15:19:54.021048+0000","last_became_active":"2024-04-25T15:19:50.517856+0000","last_became_peered":"2024-04-25T15:19:50.517856+0000","last_unstale":"2024-04-25T15:19:54.021048+0000","last_undegraded":"2024-04-25T15:19:54.021048+0000","last_fullsized":"2024-04-25T15:19:54.021048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:08:18.271358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490731+0000","last_change":"2024-04-25T15:19:46.469634+0000","last_active":"2024-04-25T15:19:52.490731+0000","last_peered":"2024-04-25T15:19:52.490731+0000","last_clean":"2024-04-25T15:19:52.490731+0000","last_became_active":"2024-04-25T15:19:46.468979+0000","last_became_peered":"2024-04-25T15:19:46.468979+0000","last_unstale":"2024-04-25T15:19:52.490731+0000","last_undegraded":"2024-04-25T15:19:52.490731+0000","last_fullsized":"2024-04-25T15:19:52.490731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017779+0000","last_change":"2024-04-25T15:19:45.471612+0000","last_active":"2024-04-25T15:19:54.017779+0000","last_peered":"2024-04-25T15:19:54.017779+0000","last_clean":"2024-04-25T15:19:54.017779+0000","last_became_active":"2024-04-25T15:19:45.471289+0000","last_became_peered":"2024-04-25T15:19:45.471289+0000","last_unstale":"2024-04-25T15:19:54.017779+0000","last_undegraded":"2024-04-25T15:19:54.017779+0000","last_fullsized":"2024-04-25T15:19:54.017779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020544+0000","last_change":"2024-04-25T15:19:51.105778+0000","last_active":"2024-04-25T15:19:54.020544+0000","last_peered":"2024-04-25T15:19:54.020544+0000","last_clean":"2024-04-25T15:19:54.020544+0000","last_became_active":"2024-04-25T15:19:51.105435+0000","last_became_peered":"2024-04-25T15:19:51.105435+0000","last_unstale":"2024-04-25T15:19:54.020544+0000","last_undegraded":"2024-04-25T15:19:54.020544+0000","last_fullsized":"2024-04-25T15:19:54.020544+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:09:15.216797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489181+0000","last_change":"2024-04-25T15:19:51.101540+0000","last_active":"2024-04-25T15:19:52.489181+0000","last_peered":"2024-04-25T15:19:52.489181+0000","last_clean":"2024-04-25T15:19:52.489181+0000","last_became_active":"2024-04-25T15:19:51.101126+0000","last_became_peered":"2024-04-25T15:19:51.101126+0000","last_unstale":"2024-04-25T15:19:52.489181+0000","last_undegraded":"2024-04-25T15:19:52.489181+0000","last_fullsized":"2024-04-25T15:19:52.489181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809353+0000","last_change":"2024-04-25T15:19:46.470144+0000","last_active":"2024-04-25T15:19:53.809353+0000","last_peered":"2024-04-25T15:19:53.809353+0000","last_clean":"2024-04-25T15:19:53.809353+0000","last_became_active":"2024-04-25T15:19:46.469786+0000","last_became_peered":"2024-04-25T15:19:46.469786+0000","last_unstale":"2024-04-25T15:19:53.809353+0000","last_undegraded":"2024-04-25T15:19:53.809353+0000","last_fullsized":"2024-04-25T15:19:53.809353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490304+0000","last_change":"2024-04-25T15:19:45.466078+0000","last_active":"2024-04-25T15:19:52.490304+0000","last_peered":"2024-04-25T15:19:52.490304+0000","last_clean":"2024-04-25T15:19:52.490304+0000","last_became_active":"2024-04-25T15:19:45.465744+0000","last_became_peered":"2024-04-25T15:19:45.465744+0000","last_unstale":"2024-04-25T15:19:52.490304+0000","last_undegraded":"2024-04-25T15:19:52.490304+0000","last_fullsized":"2024-04-25T15:19:52.490304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018744+0000","last_change":"2024-04-25T15:19:50.517045+0000","last_active":"2024-04-25T15:19:54.018744+0000","last_peered":"2024-04-25T15:19:54.018744+0000","last_clean":"2024-04-25T15:19:54.018744+0000","last_became_active":"2024-04-25T15:19:50.516571+0000","last_became_peered":"2024-04-25T15:19:50.516571+0000","last_unstale":"2024-04-25T15:19:54.018744+0000","last_undegraded":"2024-04-25T15:19:54.018744+0000","last_fullsized":"2024-04-25T15:19:54.018744+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020866+0000","last_change":"2024-04-25T15:19:46.472281+0000","last_active":"2024-04-25T15:19:54.020866+0000","last_peered":"2024-04-25T15:19:54.020866+0000","last_clean":"2024-04-25T15:19:54.020866+0000","last_became_active":"2024-04-25T15:19:46.471957+0000","last_became_peered":"2024-04-25T15:19:46.471957+0000","last_unstale":"2024-04-25T15:19:54.020866+0000","last_undegraded":"2024-04-25T15:19:54.020866+0000","last_fullsized":"2024-04-25T15:19:54.020866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021072+0000","last_change":"2024-04-25T15:19:45.461678+0000","last_active":"2024-04-25T15:19:54.021072+0000","last_peered":"2024-04-25T15:19:54.021072+0000","last_clean":"2024-04-25T15:19:54.021072+0000","last_became_active":"2024-04-25T15:19:45.461365+0000","last_became_peered":"2024-04-25T15:19:45.461365+0000","last_unstale":"2024-04-25T15:19:54.021072+0000","last_undegraded":"2024-04-25T15:19:54.021072+0000","last_fullsized":"2024-04-25T15:19:54.021072+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.468983+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.468673+0000","last_became_peered":"2024-04-25T15:19:45.468673+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489044+0000","last_change":"2024-04-25T15:19:46.459915+0000","last_active":"2024-04-25T15:19:52.489044+0000","last_peered":"2024-04-25T15:19:52.489044+0000","last_clean":"2024-04-25T15:19:52.489044+0000","last_became_active":"2024-04-25T15:19:46.459596+0000","last_became_peered":"2024-04-25T15:19:46.459596+0000","last_unstale":"2024-04-25T15:19:52.489044+0000","last_undegraded":"2024-04-25T15:19:52.489044+0000","last_fullsized":"2024-04-25T15:19:52.489044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020992+0000","last_change":"2024-04-25T15:19:50.517033+0000","last_active":"2024-04-25T15:19:54.020992+0000","last_peered":"2024-04-25T15:19:54.020992+0000","last_clean":"2024-04-25T15:19:54.020992+0000","last_became_active":"2024-04-25T15:19:50.516587+0000","last_became_peered":"2024-04-25T15:19:50.516587+0000","last_unstale":"2024-04-25T15:19:54.020992+0000","last_undegraded":"2024-04-25T15:19:54.020992+0000","last_fullsized":"2024-04-25T15:19:54.020992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:50.341053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019861+0000","last_change":"2024-04-25T15:19:45.466373+0000","last_active":"2024-04-25T15:19:54.019861+0000","last_peered":"2024-04-25T15:19:54.019861+0000","last_clean":"2024-04-25T15:19:54.019861+0000","last_became_active":"2024-04-25T15:19:45.466054+0000","last_became_peered":"2024-04-25T15:19:45.466054+0000","last_unstale":"2024-04-25T15:19:54.019861+0000","last_undegraded":"2024-04-25T15:19:54.019861+0000","last_fullsized":"2024-04-25T15:19:54.019861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:33.631245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491098+0000","last_change":"2024-04-25T15:19:46.471011+0000","last_active":"2024-04-25T15:19:52.491098+0000","last_peered":"2024-04-25T15:19:52.491098+0000","last_clean":"2024-04-25T15:19:52.491098+0000","last_became_active":"2024-04-25T15:19:46.470620+0000","last_became_peered":"2024-04-25T15:19:46.470620+0000","last_unstale":"2024-04-25T15:19:52.491098+0000","last_undegraded":"2024-04-25T15:19:52.491098+0000","last_fullsized":"2024-04-25T15:19:52.491098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:15.434932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021206+0000","last_change":"2024-04-25T15:19:50.517337+0000","last_active":"2024-04-25T15:19:54.021206+0000","last_peered":"2024-04-25T15:19:54.021206+0000","last_clean":"2024-04-25T15:19:54.021206+0000","last_became_active":"2024-04-25T15:19:50.516848+0000","last_became_peered":"2024-04-25T15:19:50.516848+0000","last_unstale":"2024-04-25T15:19:54.021206+0000","last_undegraded":"2024-04-25T15:19:54.021206+0000","last_fullsized":"2024-04-25T15:19:54.021206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:21:22.692161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018022+0000","last_change":"2024-04-25T15:19:46.476012+0000","last_active":"2024-04-25T15:19:54.018022+0000","last_peered":"2024-04-25T15:19:54.018022+0000","last_clean":"2024-04-25T15:19:54.018022+0000","last_became_active":"2024-04-25T15:19:46.475700+0000","last_became_peered":"2024-04-25T15:19:46.475700+0000","last_unstale":"2024-04-25T15:19:54.018022+0000","last_undegraded":"2024-04-25T15:19:54.018022+0000","last_fullsized":"2024-04-25T15:19:54.018022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020188+0000","last_change":"2024-04-25T15:19:46.484292+0000","last_active":"2024-04-25T15:19:54.020188+0000","last_peered":"2024-04-25T15:19:54.020188+0000","last_clean":"2024-04-25T15:19:54.020188+0000","last_became_active":"2024-04-25T15:19:46.483963+0000","last_became_peered":"2024-04-25T15:19:46.483963+0000","last_unstale":"2024-04-25T15:19:54.020188+0000","last_undegraded":"2024-04-25T15:19:54.020188+0000","last_fullsized":"2024-04-25T15:19:54.020188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490912+0000","last_change":"2024-04-25T15:19:46.484902+0000","last_active":"2024-04-25T15:19:52.490912+0000","last_peered":"2024-04-25T15:19:52.490912+0000","last_clean":"2024-04-25T15:19:52.490912+0000","last_became_active":"2024-04-25T15:19:46.482984+0000","last_became_peered":"2024-04-25T15:19:46.482984+0000","last_unstale":"2024-04-25T15:19:52.490912+0000","last_undegraded":"2024-04-25T15:19:52.490912+0000","last_fullsized":"2024-04-25T15:19:52.490912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488966+0000","last_change":"2024-04-25T15:19:46.468583+0000","last_active":"2024-04-25T15:19:52.488966+0000","last_peered":"2024-04-25T15:19:52.488966+0000","last_clean":"2024-04-25T15:19:52.488966+0000","last_became_active":"2024-04-25T15:19:46.468260+0000","last_became_peered":"2024-04-25T15:19:46.468260+0000","last_unstale":"2024-04-25T15:19:52.488966+0000","last_undegraded":"2024-04-25T15:19:52.488966+0000","last_fullsized":"2024-04-25T15:19:52.488966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021455+0000","last_change":"2024-04-25T15:19:46.478008+0000","last_active":"2024-04-25T15:19:54.021455+0000","last_peered":"2024-04-25T15:19:54.021455+0000","last_clean":"2024-04-25T15:19:54.021455+0000","last_became_active":"2024-04-25T15:19:46.477687+0000","last_became_peered":"2024-04-25T15:19:46.477687+0000","last_unstale":"2024-04-25T15:19:54.021455+0000","last_undegraded":"2024-04-25T15:19:54.021455+0000","last_fullsized":"2024-04-25T15:19:54.021455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809819+0000","last_change":"2024-04-25T15:19:46.475994+0000","last_active":"2024-04-25T15:19:53.809819+0000","last_peered":"2024-04-25T15:19:53.809819+0000","last_clean":"2024-04-25T15:19:53.809819+0000","last_became_active":"2024-04-25T15:19:46.475566+0000","last_became_peered":"2024-04-25T15:19:46.475566+0000","last_unstale":"2024-04-25T15:19:53.809819+0000","last_undegraded":"2024-04-25T15:19:53.809819+0000","last_fullsized":"2024-04-25T15:19:53.809819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018933+0000","last_change":"2024-04-25T15:19:46.480931+0000","last_active":"2024-04-25T15:19:54.018933+0000","last_peered":"2024-04-25T15:19:54.018933+0000","last_clean":"2024-04-25T15:19:54.018933+0000","last_became_active":"2024-04-25T15:19:46.480607+0000","last_became_peered":"2024-04-25T15:19:46.480607+0000","last_unstale":"2024-04-25T15:19:54.018933+0000","last_undegraded":"2024-04-25T15:19:54.018933+0000","last_fullsized":"2024-04-25T15:19:54.018933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020473+0000","last_change":"2024-04-25T15:19:46.478746+0000","last_active":"2024-04-25T15:19:54.020473+0000","last_peered":"2024-04-25T15:19:54.020473+0000","last_clean":"2024-04-25T15:19:54.020473+0000","last_became_active":"2024-04-25T15:19:46.478389+0000","last_became_peered":"2024-04-25T15:19:46.478389+0000","last_unstale":"2024-04-25T15:19:54.020473+0000","last_undegraded":"2024-04-25T15:19:54.020473+0000","last_fullsized":"2024-04-25T15:19:54.020473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490185+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.490185+0000","last_peered":"2024-04-25T15:19:52.490185+0000","last_clean":"2024-04-25T15:19:52.490185+0000","last_became_active":"2024-04-25T15:19:46.482115+0000","last_became_peered":"2024-04-25T15:19:46.482115+0000","last_unstale":"2024-04-25T15:19:52.490185+0000","last_undegraded":"2024-04-25T15:19:52.490185+0000","last_fullsized":"2024-04-25T15:19:52.490185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020755+0000","last_change":"2024-04-25T15:19:46.477554+0000","last_active":"2024-04-25T15:19:54.020755+0000","last_peered":"2024-04-25T15:19:54.020755+0000","last_clean":"2024-04-25T15:19:54.020755+0000","last_became_active":"2024-04-25T15:19:46.477240+0000","last_became_peered":"2024-04-25T15:19:46.477240+0000","last_unstale":"2024-04-25T15:19:54.020755+0000","last_undegraded":"2024-04-25T15:19:54.020755+0000","last_fullsized":"2024-04-25T15:19:54.020755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490811+0000","last_change":"2024-04-25T15:19:46.486093+0000","last_active":"2024-04-25T15:19:52.490811+0000","last_peered":"2024-04-25T15:19:52.490811+0000","last_clean":"2024-04-25T15:19:52.490811+0000","last_became_active":"2024-04-25T15:19:46.484171+0000","last_became_peered":"2024-04-25T15:19:46.484171+0000","last_unstale":"2024-04-25T15:19:52.490811+0000","last_undegraded":"2024-04-25T15:19:52.490811+0000","last_fullsized":"2024-04-25T15:19:52.490811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488875+0000","last_change":"2024-04-25T15:19:46.472207+0000","last_active":"2024-04-25T15:19:52.488875+0000","last_peered":"2024-04-25T15:19:52.488875+0000","last_clean":"2024-04-25T15:19:52.488875+0000","last_became_active":"2024-04-25T15:19:46.471780+0000","last_became_peered":"2024-04-25T15:19:46.471780+0000","last_unstale":"2024-04-25T15:19:52.488875+0000","last_undegraded":"2024-04-25T15:19:52.488875+0000","last_fullsized":"2024-04-25T15:19:52.488875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018502+0000","last_change":"2024-04-25T15:19:46.477605+0000","last_active":"2024-04-25T15:19:54.018502+0000","last_peered":"2024-04-25T15:19:54.018502+0000","last_clean":"2024-04-25T15:19:54.018502+0000","last_became_active":"2024-04-25T15:19:46.477324+0000","last_became_peered":"2024-04-25T15:19:46.477324+0000","last_unstale":"2024-04-25T15:19:54.018502+0000","last_undegraded":"2024-04-25T15:19:54.018502+0000","last_fullsized":"2024-04-25T15:19:54.018502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397007+0000","last_change":"2024-04-25T15:19:46.468508+0000","last_active":"2024-04-25T15:19:53.397007+0000","last_peered":"2024-04-25T15:19:53.397007+0000","last_clean":"2024-04-25T15:19:53.397007+0000","last_became_active":"2024-04-25T15:19:46.468170+0000","last_became_peered":"2024-04-25T15:19:46.468170+0000","last_unstale":"2024-04-25T15:19:53.397007+0000","last_undegraded":"2024-04-25T15:19:53.397007+0000","last_fullsized":"2024-04-25T15:19:53.397007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:00.962714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020658+0000","last_change":"2024-04-25T15:19:46.475011+0000","last_active":"2024-04-25T15:19:54.020658+0000","last_peered":"2024-04-25T15:19:54.020658+0000","last_clean":"2024-04-25T15:19:54.020658+0000","last_became_active":"2024-04-25T15:19:46.474637+0000","last_became_peered":"2024-04-25T15:19:46.474637+0000","last_unstale":"2024-04-25T15:19:54.020658+0000","last_undegraded":"2024-04-25T15:19:54.020658+0000","last_fullsized":"2024-04-25T15:19:54.020658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491965+0000","last_change":"2024-04-25T15:19:46.483654+0000","last_active":"2024-04-25T15:19:52.491965+0000","last_peered":"2024-04-25T15:19:52.491965+0000","last_clean":"2024-04-25T15:19:52.491965+0000","last_became_active":"2024-04-25T15:19:46.483291+0000","last_became_peered":"2024-04-25T15:19:46.483291+0000","last_unstale":"2024-04-25T15:19:52.491965+0000","last_undegraded":"2024-04-25T15:19:52.491965+0000","last_fullsized":"2024-04-25T15:19:52.491965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490226+0000","last_change":"2024-04-25T15:19:46.483513+0000","last_active":"2024-04-25T15:19:52.490226+0000","last_peered":"2024-04-25T15:19:52.490226+0000","last_clean":"2024-04-25T15:19:52.490226+0000","last_became_active":"2024-04-25T15:19:46.483158+0000","last_became_peered":"2024-04-25T15:19:46.483158+0000","last_unstale":"2024-04-25T15:19:52.490226+0000","last_undegraded":"2024-04-25T15:19:52.490226+0000","last_fullsized":"2024-04-25T15:19:52.490226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809438+0000","last_change":"2024-04-25T15:19:46.479148+0000","last_active":"2024-04-25T15:19:53.809438+0000","last_peered":"2024-04-25T15:19:53.809438+0000","last_clean":"2024-04-25T15:19:53.809438+0000","last_became_active":"2024-04-25T15:19:46.478829+0000","last_became_peered":"2024-04-25T15:19:46.478829+0000","last_unstale":"2024-04-25T15:19:53.809438+0000","last_undegraded":"2024-04-25T15:19:53.809438+0000","last_fullsized":"2024-04-25T15:19:53.809438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489010+0000","last_change":"2024-04-25T15:19:46.471858+0000","last_active":"2024-04-25T15:19:52.489010+0000","last_peered":"2024-04-25T15:19:52.489010+0000","last_clean":"2024-04-25T15:19:52.489010+0000","last_became_active":"2024-04-25T15:19:46.471517+0000","last_became_peered":"2024-04-25T15:19:46.471517+0000","last_unstale":"2024-04-25T15:19:52.489010+0000","last_undegraded":"2024-04-25T15:19:52.489010+0000","last_fullsized":"2024-04-25T15:19:52.489010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:58.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020190+0000","last_change":"2024-04-25T15:19:46.473809+0000","last_active":"2024-04-25T15:19:54.020190+0000","last_peered":"2024-04-25T15:19:54.020190+0000","last_clean":"2024-04-25T15:19:54.020190+0000","last_became_active":"2024-04-25T15:19:46.473479+0000","last_became_peered":"2024-04-25T15:19:46.473479+0000","last_unstale":"2024-04-25T15:19:54.020190+0000","last_undegraded":"2024-04-25T15:19:54.020190+0000","last_fullsized":"2024-04-25T15:19:54.020190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490638+0000","last_change":"2024-04-25T15:19:46.473643+0000","last_active":"2024-04-25T15:19:52.490638+0000","last_peered":"2024-04-25T15:19:52.490638+0000","last_clean":"2024-04-25T15:19:52.490638+0000","last_became_active":"2024-04-25T15:19:46.473190+0000","last_became_peered":"2024-04-25T15:19:46.473190+0000","last_unstale":"2024-04-25T15:19:52.490638+0000","last_undegraded":"2024-04-25T15:19:52.490638+0000","last_fullsized":"2024-04-25T15:19:52.490638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018544+0000","last_change":"2024-04-25T15:19:46.488270+0000","last_active":"2024-04-25T15:19:54.018544+0000","last_peered":"2024-04-25T15:19:54.018544+0000","last_clean":"2024-04-25T15:19:54.018544+0000","last_became_active":"2024-04-25T15:19:46.487935+0000","last_became_peered":"2024-04-25T15:19:46.487935+0000","last_unstale":"2024-04-25T15:19:54.018544+0000","last_undegraded":"2024-04-25T15:19:54.018544+0000","last_fullsized":"2024-04-25T15:19:54.018544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018847+0000","last_change":"2024-04-25T15:19:46.488350+0000","last_active":"2024-04-25T15:19:54.018847+0000","last_peered":"2024-04-25T15:19:54.018847+0000","last_clean":"2024-04-25T15:19:54.018847+0000","last_became_active":"2024-04-25T15:19:46.487997+0000","last_became_peered":"2024-04-25T15:19:46.487997+0000","last_unstale":"2024-04-25T15:19:54.018847+0000","last_undegraded":"2024-04-25T15:19:54.018847+0000","last_fullsized":"2024-04-25T15:19:54.018847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397147+0000","last_change":"2024-04-25T15:19:46.486142+0000","last_active":"2024-04-25T15:19:53.397147+0000","last_peered":"2024-04-25T15:19:53.397147+0000","last_clean":"2024-04-25T15:19:53.397147+0000","last_became_active":"2024-04-25T15:19:46.485774+0000","last_became_peered":"2024-04-25T15:19:46.485774+0000","last_unstale":"2024-04-25T15:19:53.397147+0000","last_undegraded":"2024-04-25T15:19:53.397147+0000","last_fullsized":"2024-04-25T15:19:53.397147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:16.297767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020777+0000","last_change":"2024-04-25T15:19:46.486137+0000","last_active":"2024-04-25T15:19:54.020777+0000","last_peered":"2024-04-25T15:19:54.020777+0000","last_clean":"2024-04-25T15:19:54.020777+0000","last_became_active":"2024-04-25T15:19:46.485814+0000","last_became_peered":"2024-04-25T15:19:46.485814+0000","last_unstale":"2024-04-25T15:19:54.020777+0000","last_undegraded":"2024-04-25T15:19:54.020777+0000","last_fullsized":"2024-04-25T15:19:54.020777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.469644+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469320+0000","last_became_peered":"2024-04-25T15:19:46.469320+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490739+0000","last_change":"2024-04-25T15:19:46.486357+0000","last_active":"2024-04-25T15:19:52.490739+0000","last_peered":"2024-04-25T15:19:52.490739+0000","last_clean":"2024-04-25T15:19:52.490739+0000","last_became_active":"2024-04-25T15:19:46.486016+0000","last_became_peered":"2024-04-25T15:19:46.486016+0000","last_unstale":"2024-04-25T15:19:52.490739+0000","last_undegraded":"2024-04-25T15:19:52.490739+0000","last_fullsized":"2024-04-25T15:19:52.490739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.470290+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469965+0000","last_became_peered":"2024-04-25T15:19:46.469965+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491250+0000","last_change":"2024-04-25T15:19:46.483532+0000","last_active":"2024-04-25T15:19:52.491250+0000","last_peered":"2024-04-25T15:19:52.491250+0000","last_clean":"2024-04-25T15:19:52.491250+0000","last_became_active":"2024-04-25T15:19:46.483242+0000","last_became_peered":"2024-04-25T15:19:46.483242+0000","last_unstale":"2024-04-25T15:19:52.491250+0000","last_undegraded":"2024-04-25T15:19:52.491250+0000","last_fullsized":"2024-04-25T15:19:52.491250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489031+0000","last_change":"2024-04-25T15:19:46.467386+0000","last_active":"2024-04-25T15:19:52.489031+0000","last_peered":"2024-04-25T15:19:52.489031+0000","last_clean":"2024-04-25T15:19:52.489031+0000","last_became_active":"2024-04-25T15:19:46.467034+0000","last_became_peered":"2024-04-25T15:19:46.467034+0000","last_unstale":"2024-04-25T15:19:52.489031+0000","last_undegraded":"2024-04-25T15:19:52.489031+0000","last_fullsized":"2024-04-25T15:19:52.489031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809963+0000","last_change":"2024-04-25T15:19:46.472318+0000","last_active":"2024-04-25T15:19:53.809963+0000","last_peered":"2024-04-25T15:19:53.809963+0000","last_clean":"2024-04-25T15:19:53.809963+0000","last_became_active":"2024-04-25T15:19:46.471982+0000","last_became_peered":"2024-04-25T15:19:46.471982+0000","last_unstale":"2024-04-25T15:19:53.809963+0000","last_undegraded":"2024-04-25T15:19:53.809963+0000","last_fullsized":"2024-04-25T15:19:53.809963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397045+0000","last_change":"2024-04-25T15:19:46.473366+0000","last_active":"2024-04-25T15:19:53.397045+0000","last_peered":"2024-04-25T15:19:53.397045+0000","last_clean":"2024-04-25T15:19:53.397045+0000","last_became_active":"2024-04-25T15:19:46.472969+0000","last_became_peered":"2024-04-25T15:19:46.472969+0000","last_unstale":"2024-04-25T15:19:53.397045+0000","last_undegraded":"2024-04-25T15:19:53.397045+0000","last_fullsized":"2024-04-25T15:19:53.397045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:20:09.904 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:20:09.906 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T15:20:09.906 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T15:20:09.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T15:20:09.907 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T15:20:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.048+0000 7fdcb553f640 1 Processor -- start 2024-04-25T15:20:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.048+0000 7fdcb553f640 1 -- start start 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcb553f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 0x7fdcb00d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcb553f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 0x7fdcb006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcb553f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb00b9200 con 0x7fdcb00d4f60 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcb553f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fdcb006c550 con 0x7fdcb006ccc0 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcb553f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb006c830 con 0x7fdcb00d0730 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcae7fc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 0x7fdcb00d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcaf7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 0x7fdcb006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcaf7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 0x7fdcb006d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38914/0 (socket says 172.21.15.146:38914) 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcaeffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fdcb00d4f60 0x7fdcb00d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:52236/0 (socket says 172.21.15.146:52236) 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcaf7fe640 1 -- 172.21.15.146:0/1173570150 learned_addr learned my addr 172.21.15.146:0/1173570150 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:10.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcae7fc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 0x7fdcb00d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:54762/0 (socket says 172.21.15.146:54762) 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcae7fc640 1 -- 172.21.15.146:0/1173570150 >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 msgr2=0x7fdcb006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.050+0000 7fdcae7fc640 1 --2- 172.21.15.146:0/1173570150 >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 0x7fdcb006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcae7fc640 1 -- 172.21.15.146:0/1173570150 >> v1:172.21.15.177:6789/0 conn(0x7fdcb00d4f60 legacy=0x7fdcb00d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcae7fc640 1 -- 172.21.15.146:0/1173570150 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb00d1070 con 0x7fdcb00d0730 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcaf7fe640 1 --2- 172.21.15.146:0/1173570150 >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 0x7fdcb006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcae7fc640 1 --2- 172.21.15.146:0/1173570150 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 0x7fdcb00d5370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdca40097a0 tx=0x7fdca402a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f764653de2d7cdd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcadffb640 1 -- 172.21.15.146:0/1173570150 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4037070 con 0x7fdcb00d0730 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcadffb640 1 -- 172.21.15.146:0/1173570150 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca402adb0 con 0x7fdcb00d0730 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.051+0000 7fdcadffb640 1 -- 172.21.15.146:0/1173570150 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4032660 con 0x7fdcb00d0730 2024-04-25T15:20:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 -- 172.21.15.146:0/1173570150 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 msgr2=0x7fdcb00d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 --2- 172.21.15.146:0/1173570150 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 0x7fdcb00d5370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdca40097a0 tx=0x7fdca402a060 comp rx=0 tx=0).stop 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 -- 172.21.15.146:0/1173570150 shutdown_connections 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 --2- 172.21.15.146:0/1173570150 >> v2:172.21.15.177:3300/0 conn(0x7fdcb006ccc0 0x7fdcb006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 --2- 172.21.15.146:0/1173570150 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00d0730 0x7fdcb00d5370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 -- 172.21.15.146:0/1173570150 >> 172.21.15.146:0/1173570150 conn(0x7fdcb00d2450 msgr2=0x7fdcb00d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 -- 172.21.15.146:0/1173570150 shutdown_connections 2024-04-25T15:20:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.052+0000 7fdcb553f640 1 -- 172.21.15.146:0/1173570150 wait complete. 2024-04-25T15:20:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.054+0000 7fdcb553f640 1 Processor -- start 2024-04-25T15:20:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.054+0000 7fdcb553f640 1 -- start start 2024-04-25T15:20:10.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.055+0000 7fdcb553f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006ccc0 0x7fdcb00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.055+0000 7fdcb553f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 0x7fdcb0162250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.055+0000 7fdcb553f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00cc700 0x7fdcb0162790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.055+0000 7fdcb553f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb00b5150 con 0x7fdcb00d4f60 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.055+0000 7fdcb553f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb00d15f0 con 0x7fdcb006ccc0 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.055+0000 7fdcb553f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb00b52d0 con 0x7fdcb00cc700 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 0x7fdcb0162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 0x7fdcb0162250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36854/0 (socket says 172.21.15.146:36854) 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 -- 172.21.15.146:0/3716755103 learned_addr learned my addr 172.21.15.146:0/3716755103 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcaeffd640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006ccc0 0x7fdcb00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcaf7fe640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00cc700 0x7fdcb0162790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 -- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00cc700 msgr2=0x7fdcb0162790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00cc700 0x7fdcb0162790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 -- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006ccc0 msgr2=0x7fdcb00cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006ccc0 0x7fdcb00cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 -- 172.21.15.146:0/3716755103 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb0160840 con 0x7fdcb00d4f60 2024-04-25T15:20:10.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcae7fc640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 0x7fdcb0162250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdca402a0a0 tx=0x7fdca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4041020 con 0x7fdcb00d4f60 2024-04-25T15:20:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca403c670 con 0x7fdcb00d4f60 2024-04-25T15:20:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb0160b20 con 0x7fdcb00d4f60 2024-04-25T15:20:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4037040 con 0x7fdcb00d4f60 2024-04-25T15:20:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.056+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb01610e0 con 0x7fdcb00d4f60 2024-04-25T15:20:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.058+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb006a0e0 con 0x7fdcb00d4f60 2024-04-25T15:20:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.058+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdca4044020 con 0x7fdcb00d4f60 2024-04-25T15:20:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.058+0000 7fdc977fe640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc9c0744e0 0x7fdc9c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.058+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdca40b5b40 con 0x7fdcb00d4f60 2024-04-25T15:20:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.058+0000 7fdcaeffd640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc9c0744e0 0x7fdc9c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.059+0000 7fdcaeffd640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc9c0744e0 0x7fdc9c0769f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdca0007fd0 tx=0x7fdca0007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.063+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdca404b030 con 0x7fdcb00d4f60 2024-04-25T15:20:10.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.284+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fdcb00bb1e0 con 0x7fdcb00d4f60 2024-04-25T15:20:10.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.287+0000 7fdc977fe640 1 -- 172.21.15.146:0/3716755103 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+186750 (secure 0 0 0) 0x7fdca407f9d0 con 0x7fdcb00d4f60 2024-04-25T15:20:10.292 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc9c0744e0 msgr2=0x7fdc9c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc9c0744e0 0x7fdc9c0769f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdca0007fd0 tx=0x7fdca0007e80 comp rx=0 tx=0).stop 2024-04-25T15:20:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 msgr2=0x7fdcb0162250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 0x7fdcb0162250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdca402a0a0 tx=0x7fdca40046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 shutdown_connections 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc9c0744e0 0x7fdc9c0769f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00cc700 0x7fdcb0162790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00d4f60 0x7fdcb0162250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 --2- 172.21.15.146:0/3716755103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006ccc0 0x7fdcb00cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 >> 172.21.15.146:0/3716755103 conn(0x7fdcb00d2450 msgr2=0x7fdcb00b93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 shutdown_connections 2024-04-25T15:20:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.290+0000 7fdcb553f640 1 -- 172.21.15.146:0/3716755103 wait complete. 2024-04-25T15:20:10.302 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1999496207},{"type":"v1","addr":"172.21.15.146:6833","nonce":1999496207}]},"active_addr":"172.21.15.146:6833/1999496207","active_change":"2024-04-25T15:19:25.976738+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":1353683041}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3437434691}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2310867839}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":1180258365}]}]} 2024-04-25T15:20:10.306 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T15:20:10.306 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T15:20:10.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:10.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.413+0000 7f807b870640 1 Processor -- start 2024-04-25T15:20:10.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.413+0000 7f807b870640 1 -- start start 2024-04-25T15:20:10.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807b870640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80740d21f0 0x7f807415ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807b870640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f807406f6d0 0x7f80740b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807b870640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f807409c2a0 con 0x7f807406f2c0 2024-04-25T15:20:10.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807b870640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f80740d1d60 con 0x7f80740d21f0 2024-04-25T15:20:10.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807b870640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f807409bce0 con 0x7f807406f6d0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807960c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80740d21f0 0x7f807415ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807960c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80740d21f0 0x7f807415ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36858/0 (socket says 172.21.15.146:36858) 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.414+0000 7f807960c640 1 -- 172.21.15.146:0/3497201624 learned_addr learned my addr 172.21.15.146:0/3497201624 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.415+0000 7f8079e0d640 1 --2- 172.21.15.146:0/3497201624 >> v2:172.21.15.177:3300/0 conn(0x7f807406f6d0 0x7f80740b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.415+0000 7f807960c640 1 -- 172.21.15.146:0/3497201624 >> v2:172.21.15.177:3300/0 conn(0x7f807406f6d0 msgr2=0x7f80740b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.415+0000 7f807960c640 1 --2- 172.21.15.146:0/3497201624 >> v2:172.21.15.177:3300/0 conn(0x7f807406f6d0 0x7f80740b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.415+0000 7f807960c640 1 -- 172.21.15.146:0/3497201624 >> v1:172.21.15.177:6789/0 conn(0x7f807406f2c0 legacy=0x7f80740b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.415+0000 7f807960c640 1 -- 172.21.15.146:0/3497201624 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807409bfc0 con 0x7f80740d21f0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.415+0000 7f807960c640 1 --2- 172.21.15.146:0/3497201624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80740d21f0 0x7f807415ded0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80740d00a0 tx=0x7f807002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4b7f3c3bf02035 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f806bfff640 1 -- 172.21.15.146:0/3497201624 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070037070 con 0x7f80740d21f0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f806bfff640 1 -- 172.21.15.146:0/3497201624 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f807002ae50 con 0x7f80740d21f0 2024-04-25T15:20:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f806bfff640 1 -- 172.21.15.146:0/3497201624 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070032860 con 0x7f80740d21f0 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 -- 172.21.15.146:0/3497201624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80740d21f0 msgr2=0x7f807415ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 --2- 172.21.15.146:0/3497201624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80740d21f0 0x7f807415ded0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80740d00a0 tx=0x7f807002a300 comp rx=0 tx=0).stop 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 -- 172.21.15.146:0/3497201624 shutdown_connections 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 --2- 172.21.15.146:0/3497201624 >> v2:172.21.15.177:3300/0 conn(0x7f807406f6d0 0x7f80740b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 --2- 172.21.15.146:0/3497201624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80740d21f0 0x7f807415ded0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 -- 172.21.15.146:0/3497201624 >> 172.21.15.146:0/3497201624 conn(0x7f80740b7530 msgr2=0x7f80740b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 -- 172.21.15.146:0/3497201624 shutdown_connections 2024-04-25T15:20:10.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.416+0000 7f807b870640 1 -- 172.21.15.146:0/3497201624 wait complete. 2024-04-25T15:20:10.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.418+0000 7f807b870640 1 Processor -- start 2024-04-25T15:20:10.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.418+0000 7f807b870640 1 -- start start 2024-04-25T15:20:10.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.419+0000 7f807b870640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807406f2c0 0x7f80741544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.419+0000 7f807b870640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 0x7f807415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.419+0000 7f807b870640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807415cee0 0x7f8074158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.419+0000 7f807b870640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f807409b580 con 0x7f807406f6d0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.419+0000 7f807b870640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f807409b400 con 0x7f807406f2c0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.419+0000 7f807b870640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f80740b3de0 con 0x7f807415cee0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 0x7f807415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 0x7f807415c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36862/0 (socket says 172.21.15.146:36862) 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 -- 172.21.15.146:0/703215492 learned_addr learned my addr 172.21.15.146:0/703215492 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f807960c640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807406f2c0 0x7f80741544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8079e0d640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807415cee0 0x7f8074158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 -- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807415cee0 msgr2=0x7f8074158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807415cee0 0x7f8074158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 -- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807406f2c0 msgr2=0x7f80741544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807406f2c0 0x7f80741544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 -- 172.21.15.146:0/703215492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807406d130 con 0x7f807406f6d0 2024-04-25T15:20:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8078e0b640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 0x7f807415c9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f806400c470 tx=0x7f806400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640052d0 con 0x7f807406f6d0 2024-04-25T15:20:10.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8064005490 con 0x7f807406f6d0 2024-04-25T15:20:10.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807406d410 con 0x7f807406f6d0 2024-04-25T15:20:10.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8064007260 con 0x7f807406f6d0 2024-04-25T15:20:10.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.420+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807406d9d0 con 0x7f807406f6d0 2024-04-25T15:20:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.422+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807406ce90 con 0x7f807406f6d0 2024-04-25T15:20:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.422+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8064007400 con 0x7f807406f6d0 2024-04-25T15:20:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.422+0000 7f8069ffb640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80580744e0 0x7f80580769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.422+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8064094930 con 0x7f807406f6d0 2024-04-25T15:20:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.422+0000 7f807960c640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80580744e0 0x7f80580769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.423+0000 7f807960c640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80580744e0 0x7f80580769f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8070034000 tx=0x7f807002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.427+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f807406ce90 con 0x7f807406f6d0 2024-04-25T15:20:10.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.605+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f807406a0e0 con 0x7f807406f6d0 2024-04-25T15:20:10.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.606+0000 7f8069ffb640 1 -- 172.21.15.146:0/703215492 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7f806405e730 con 0x7f807406f6d0 2024-04-25T15:20:10.607 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:10.607 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:10.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80580744e0 msgr2=0x7f80580769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80580744e0 0x7f80580769f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8070034000 tx=0x7f807002e000 comp rx=0 tx=0).stop 2024-04-25T15:20:10.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 msgr2=0x7f807415c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 0x7f807415c9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f806400c470 tx=0x7f806400c930 comp rx=0 tx=0).stop 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 shutdown_connections 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80580744e0 0x7f80580769f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807415cee0 0x7f8074158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f807406f6d0 0x7f807415c9a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 --2- 172.21.15.146:0/703215492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807406f2c0 0x7f80741544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 >> 172.21.15.146:0/703215492 conn(0x7f80740b7530 msgr2=0x7f80740d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 shutdown_connections 2024-04-25T15:20:10.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.608+0000 7f807b870640 1 -- 172.21.15.146:0/703215492 wait complete. 2024-04-25T15:20:10.621 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T15:20:10.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:10.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.728+0000 7ffb7a96c640 1 Processor -- start 2024-04-25T15:20:10.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.728+0000 7ffb7a96c640 1 -- start start 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb7a96c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb740b5950 0x7ffb740d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb7a96c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb740d2230 con 0x7ffb7415ffa0 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb7a96c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb740cfa80 con 0x7ffb740b6610 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb7a96c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffb740b2f50 con 0x7ffb740b5950 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb737fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb740b5950 0x7ffb740d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb737fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb740b5950 0x7ffb740d2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59854/0 (socket says 172.21.15.146:59854) 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb78f09640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffb7415ffa0 0x7ffb740b7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40538/0 (socket says 172.21.15.146:40538) 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb737fe640 1 -- 172.21.15.146:0/3363021886 learned_addr learned my addr 172.21.15.146:0/3363021886 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:10.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb737fe640 1 -- 172.21.15.146:0/3363021886 >> v1:172.21.15.177:6789/0 conn(0x7ffb740b6610 legacy=0x7ffb740b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb737fe640 1 -- 172.21.15.146:0/3363021886 >> v1:172.21.15.146:6789/0 conn(0x7ffb7415ffa0 legacy=0x7ffb740b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb737fe640 1 -- 172.21.15.146:0/3363021886 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb740cfd60 con 0x7ffb740b5950 2024-04-25T15:20:10.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.730+0000 7ffb72ffd640 1 -- 172.21.15.146:0/3363021886 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075421370 0 0) 0x7ffb740d2230 con 0x7ffb7415ffa0 2024-04-25T15:20:10.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.731+0000 7ffb737fe640 1 --2- 172.21.15.146:0/3363021886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb740b5950 0x7ffb740d2df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009870 tx=0x7ffb6802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93159f6094d0d5cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.731+0000 7ffb72ffd640 1 -- 172.21.15.146:0/3363021886 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68037070 con 0x7ffb740b5950 2024-04-25T15:20:10.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.731+0000 7ffb72ffd640 1 -- 172.21.15.146:0/3363021886 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb6802ae50 con 0x7ffb740b5950 2024-04-25T15:20:10.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.731+0000 7ffb72ffd640 1 -- 172.21.15.146:0/3363021886 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68032860 con 0x7ffb740b5950 2024-04-25T15:20:10.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 -- 172.21.15.146:0/3363021886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb740b5950 msgr2=0x7ffb740d2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/3363021886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb740b5950 0x7ffb740d2df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009870 tx=0x7ffb6802a300 comp rx=0 tx=0).stop 2024-04-25T15:20:10.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 -- 172.21.15.146:0/3363021886 shutdown_connections 2024-04-25T15:20:10.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/3363021886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb740b5950 0x7ffb740d2df0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 -- 172.21.15.146:0/3363021886 >> 172.21.15.146:0/3363021886 conn(0x7ffb740d0240 msgr2=0x7ffb740d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:10.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 -- 172.21.15.146:0/3363021886 shutdown_connections 2024-04-25T15:20:10.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.732+0000 7ffb7a96c640 1 -- 172.21.15.146:0/3363021886 wait complete. 2024-04-25T15:20:10.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.733+0000 7ffb7a96c640 1 Processor -- start 2024-04-25T15:20:10.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.734+0000 7ffb7a96c640 1 -- start start 2024-04-25T15:20:10.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb7a96c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb740b6610 0x7ffb74151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb7a96c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 0x7ffb741575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb7a96c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb74153160 0x7ffb74153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb7a96c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffb740d1da0 con 0x7ffb7415ffa0 2024-04-25T15:20:10.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb7a96c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb740b2ac0 con 0x7ffb74153160 2024-04-25T15:20:10.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb7a96c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffb740d1f20 con 0x7ffb740b6610 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 0x7ffb741575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 0x7ffb741575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59870/0 (socket says 172.21.15.146:59870) 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 -- 172.21.15.146:0/925009372 learned_addr learned my addr 172.21.15.146:0/925009372 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb78f09640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb74153160 0x7ffb74153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb73fff640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb740b6610 0x7ffb74151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 -- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb740b6610 msgr2=0x7ffb74151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb740b6610 0x7ffb74151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 -- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb74153160 msgr2=0x7ffb74153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb74153160 0x7ffb74153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 -- 172.21.15.146:0/925009372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb74153ab0 con 0x7ffb7415ffa0 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb73fff640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb740b6610 0x7ffb74151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:10.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb78f09640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb74153160 0x7ffb74153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.735+0000 7ffb737fe640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 0x7ffb741575e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffb68000c00 tx=0x7ffb680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.736+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68041070 con 0x7ffb7415ffa0 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.736+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb6803c870 con 0x7ffb7415ffa0 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.736+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7406f300 con 0x7ffb7415ffa0 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.736+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68037040 con 0x7ffb7415ffa0 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.736+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb7406f8c0 con 0x7ffb7415ffa0 2024-04-25T15:20:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.737+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7406f050 con 0x7ffb7415ffa0 2024-04-25T15:20:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.737+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffb68044070 con 0x7ffb7415ffa0 2024-04-25T15:20:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.738+0000 7ffb70ff9640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb600744e0 0x7ffb600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.738+0000 7ffb73fff640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb600744e0 0x7ffb600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.738+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ffb680b5eb0 con 0x7ffb7415ffa0 2024-04-25T15:20:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.738+0000 7ffb73fff640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb600744e0 0x7ffb600769f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffb64007fd0 tx=0x7ffb64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:10.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.742+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb74153ab0 con 0x7ffb7415ffa0 2024-04-25T15:20:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.917+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb7406a0e0 con 0x7ffb7415ffa0 2024-04-25T15:20:10.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.918+0000 7ffb70ff9640 1 -- 172.21.15.146:0/925009372 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7ffb68080390 con 0x7ffb7415ffa0 2024-04-25T15:20:10.919 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:10.919 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:10.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb600744e0 msgr2=0x7ffb600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb600744e0 0x7ffb600769f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffb64007fd0 tx=0x7ffb64009040 comp rx=0 tx=0).stop 2024-04-25T15:20:10.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 msgr2=0x7ffb741575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:10.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 0x7ffb741575e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffb68000c00 tx=0x7ffb680046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 shutdown_connections 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb600744e0 0x7ffb600769f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb74153160 0x7ffb74153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7415ffa0 0x7ffb741575e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 --2- 172.21.15.146:0/925009372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb740b6610 0x7ffb74151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 >> 172.21.15.146:0/925009372 conn(0x7ffb740d0240 msgr2=0x7ffb74163f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 shutdown_connections 2024-04-25T15:20:10.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:10.920+0000 7ffb7a96c640 1 -- 172.21.15.146:0/925009372 wait complete. 2024-04-25T15:20:10.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T15:20:10.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T15:20:10.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T15:20:10.934 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T15:20:10.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T15:20:10.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T15:20:10.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T15:20:10.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T15:20:11.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.146+0000 7f8e94c35640 1 Processor -- start 2024-04-25T15:20:11.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.146+0000 7f8e94c35640 1 -- start start 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.148+0000 7f8e94c35640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e88006950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.148+0000 7f8e94c35640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e88053ad0 con 0x7f8e88050440 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.148+0000 7f8e94c35640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e88053db0 con 0x7f8e88105610 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.148+0000 7f8e94c35640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8e88054090 con 0x7f8e88105a20 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8ffff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e88006950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8effd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f8e88105610 0x7f8e88006240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38900/0 (socket says 172.21.15.146:38900) 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8effd640 1 -- 172.21.15.146:0/180515553 learned_addr learned my addr 172.21.15.146:0/180515553 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4032955917 0 0) 0x7f8e88053db0 con 0x7f8e88105610 2024-04-25T15:20:11.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e80003520 con 0x7f8e88105610 2024-04-25T15:20:11.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2992884322 0 0) 0x7f8e80003520 con 0x7f8e88105610 2024-04-25T15:20:11.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 msgr2=0x7f8e88006950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8e7fc640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e88006950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.149+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 >> v1:172.21.15.146:6789/0 conn(0x7f8e88050440 legacy=0x7f8e88104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e88054370 con 0x7f8e88105610 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f8e84000ed0 con 0x7f8e88105610 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 >> v1:172.21.15.177:6789/0 conn(0x7f8e88105610 legacy=0x7f8e88006240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e800039b0 0x7f8e80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e80004800 0x7f8e80004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e800052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88054090 con 0x7f8e800039b0 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8effd640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e80004800 0x7f8e80004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8000bb80 con 0x7f8e80004800 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8000bd80 con 0x7f8e88105a20 2024-04-25T15:20:11.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.150+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e800052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f6611f34640 1 Processor -- start 2024-04-25T15:20:11.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8f7fe640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e800039b0 0x7f8e80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8ffff640 1 -- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e80004800 msgr2=0x7f8e80004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e80004800 0x7f8e80004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8ffff640 1 -- 172.21.15.146:0/180515553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e800039b0 msgr2=0x7f8e80003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f5b52be3640 1 Processor -- start 2024-04-25T15:20:11.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e800039b0 0x7f8e80003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8ffff640 1 -- 172.21.15.146:0/180515553 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e80003520 con 0x7f8e88105a20 2024-04-25T15:20:11.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f8e8f7fe640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e800039b0 0x7f8e80003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7ff0bf413640 1 Processor -- start 2024-04-25T15:20:11.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.152+0000 7ff0bf413640 1 -- start start 2024-04-25T15:20:11.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.153+0000 7f58220d8640 1 Processor -- start 2024-04-25T15:20:11.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.153+0000 7f58220d8640 1 -- start start 2024-04-25T15:20:11.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.153+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e800052d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c010e80 tx=0x7f8e80005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1a21f4deee50c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.153+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c010a80 con 0x7f8e88105a20 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.153+0000 7f8e8e7fc640 1 -- 172.21.15.146:0/180515553 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e7c039070 con 0x7f8e88105a20 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f5b52be3640 1 -- start start 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b52be3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5b44048100 0x7f5b44048630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b52be3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b44050240 con 0x7f5b440eb6a0 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b52be3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b44050520 con 0x7f5b44006aa0 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b52be3640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5b44006580 con 0x7f5b44048100 2024-04-25T15:20:11.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b51180640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5b44006aa0 0x7f5b44006eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38902/0 (socket says 172.21.15.146:38902) 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b51180640 1 -- 172.21.15.146:0/476944920 learned_addr learned my addr 172.21.15.146:0/476944920 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.151+0000 7f6611f34640 1 -- start start 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f6611f34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c0d4f50 0x7f660c164370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f6611f34640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660c0d3ef0 con 0x7f660c0b4e50 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f6611f34640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660c1648b0 con 0x7f660c0d0af0 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f6611f34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0d4190 con 0x7f660c0d4f50 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f660bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f660c0b4e50 0x7f660c0d03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40552/0 (socket says 172.21.15.146:40552) 2024-04-25T15:20:11.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f660bfff640 1 -- 172.21.15.146:0/3453052840 learned_addr learned my addr 172.21.15.146:0/3453052840 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bf413640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff0b80d2580 0x7ff0b80b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bf413640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0b806c550 0x7ff0b806c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bf413640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b80d2990 con 0x7ff0b815cc20 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bf413640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b80d2c20 con 0x7ff0b806c550 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bf413640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b80b1d90 con 0x7ff0b80d2580 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b5097f640 1 --2- 172.21.15.146:0/476944920 >> v2:172.21.15.177:3300/0 conn(0x7f5b44048100 0x7f5b44048630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897155417 0 0) 0x7f5b44050520 con 0x7f5b44006aa0 2024-04-25T15:20:11.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b3c003520 con 0x7f5b44006aa0 2024-04-25T15:20:11.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f8e94c35640 1 -- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 msgr2=0x7f8e800052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7f8e94c35640 1 --2- 172.21.15.146:0/180515553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e88105a20 0x7f8e800052d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c010e80 tx=0x7f8e80005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f8e8ffff640 1 -- 172.21.15.146:0/180515553 reap_dead start 2024-04-25T15:20:11.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f8e94c35640 1 -- 172.21.15.146:0/180515553 shutdown_connections 2024-04-25T15:20:11.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f8e94c35640 1 -- 172.21.15.146:0/180515553 >> 172.21.15.146:0/180515553 conn(0x7f8e88054cf0 msgr2=0x7f8e88055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bd9b0640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff0b815cc20 0x7ff0b815d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40558/0 (socket says 172.21.15.146:40558) 2024-04-25T15:20:11.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.154+0000 7ff0bd9b0640 1 -- 172.21.15.146:0/994225125 learned_addr learned my addr 172.21.15.146:0/994225125 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f6610f32640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c0d4f50 0x7f660c164370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3882515986 0 0) 0x7f5b3c003520 con 0x7f5b44006aa0 2024-04-25T15:20:11.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775827445 0 0) 0x7f660c0d3ef0 con 0x7f660c0b4e50 2024-04-25T15:20:11.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65f4003520 con 0x7f660c0b4e50 2024-04-25T15:20:11.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804703978 0 0) 0x7f660c1648b0 con 0x7f660c0d0af0 2024-04-25T15:20:11.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f660c0d3ef0 con 0x7f660c0d0af0 2024-04-25T15:20:11.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7ff0bd1af640 1 --2- 172.21.15.146:0/994225125 >> v2:172.21.15.177:3300/0 conn(0x7ff0b80d2580 0x7ff0b80b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 >> v2:172.21.15.177:3300/0 conn(0x7f5b44048100 msgr2=0x7f5b44048630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 --2- 172.21.15.146:0/476944920 >> v2:172.21.15.177:3300/0 conn(0x7f5b44048100 0x7f5b44048630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 >> v1:172.21.15.146:6789/0 conn(0x7f5b440eb6a0 legacy=0x7f5b440ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/994225125 >> v2:172.21.15.146:3300/0 conn(0x7ff0b806c550 0x7ff0b806c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4127545775 0 0) 0x7f660c0d3ef0 con 0x7f660c0d0af0 2024-04-25T15:20:11.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c0d4f50 msgr2=0x7f660c164370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b440ec1c0 con 0x7f5b44006aa0 2024-04-25T15:20:11.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f5b40001670 con 0x7f5b44006aa0 2024-04-25T15:20:11.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 >> v1:172.21.15.177:6789/0 conn(0x7f5b44006aa0 legacy=0x7f5b44006eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c0d4f50 0x7f660c164370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f5b4b7fe640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b3c0039b0 0x7f5b3c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f8e94c35640 1 -- 172.21.15.146:0/180515553 shutdown_connections 2024-04-25T15:20:11.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3789273475 0 0) 0x7ff0b80d2990 con 0x7ff0b815cc20 2024-04-25T15:20:11.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0a4003520 con 0x7ff0b815cc20 2024-04-25T15:20:11.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2016750973 0 0) 0x7ff0a4003520 con 0x7ff0b815cc20 2024-04-25T15:20:11.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 >> v2:172.21.15.177:3300/0 conn(0x7ff0b80d2580 msgr2=0x7ff0b80b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 --2- 172.21.15.146:0/994225125 >> v2:172.21.15.177:3300/0 conn(0x7ff0b80d2580 0x7ff0b80b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 >> v2:172.21.15.146:3300/0 conn(0x7ff0b806c550 msgr2=0x7ff0b806c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 --2- 172.21.15.146:0/994225125 >> v2:172.21.15.146:3300/0 conn(0x7ff0b806c550 0x7ff0b806c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b8162030 con 0x7ff0b815cc20 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7ff0a8001670 con 0x7ff0b815cc20 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 >> v1:172.21.15.146:6789/0 conn(0x7ff0b815cc20 legacy=0x7ff0b815d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0a4003a70 0x7ff0a4003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0a4004800 0x7ff0a4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a40051a0 0x7ff0a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80d2c20 con 0x7ff0a4003a70 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80b1d90 con 0x7ff0a40051a0 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a400c1c0 con 0x7ff0a4004800 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/994225125 >> v2:172.21.15.146:3300/0 conn(0x7ff0b806c550 0x7ff0b806c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0a4004800 0x7ff0a4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7ff0bd9b0640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a40051a0 0x7ff0a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7fdab0f10640 1 Processor -- start 2024-04-25T15:20:11.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7fdab0f10640 1 -- start start 2024-04-25T15:20:11.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b4b7fe640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c004800 0x7f5b3c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b4b7fe640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b3c005150 0x7f5b3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b51180640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b3c005150 0x7f5b3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b5097f640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b3c0039b0 0x7f5b3c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44006580 con 0x7f5b3c005150 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b4bfff640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c004800 0x7f5b3c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c00c0c0 con 0x7f5b3c004800 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5b3c00c2c0 con 0x7f5b3c0039b0 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 >> v1:172.21.15.146:6789/0 conn(0x7f660c0b4e50 legacy=0x7f660c0d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c0d4470 con 0x7f660c0d0af0 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f65fc001690 con 0x7f660c0d0af0 2024-04-25T15:20:11.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 >> v1:172.21.15.177:6789/0 conn(0x7f660c0d0af0 legacy=0x7f660c15f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65f4003b40 0x7f65f4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65f4004800 0x7f65f4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4005150 0x7f65f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0d4190 con 0x7f65f4005150 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f65f400c060 con 0x7f65f4003b40 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.155+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f65f400c260 con 0x7f65f4004800 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f660bfff640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65f4004800 0x7f65f4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f6610f32640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65f4003b40 0x7f65f4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f660bfff640 1 -- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65f4003b40 msgr2=0x7f65f4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f58220d8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f581404ec80 0x7f58140060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f58220d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58140e5b60 0x7f5814038560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f58220d8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5814038aa0 0x7f5814050780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f58220d8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f581403ad20 con 0x7f5814038aa0 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f58220d8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5814038eb0 con 0x7f581404ec80 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f58220d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f581403a760 con 0x7f58140e5b60 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f581b7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f581404ec80 0x7f58140060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f581b7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f581404ec80 0x7f58140060c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45274/0 (socket says 172.21.15.146:45274) 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f581b7fe640 1 -- 172.21.15.146:0/374555103 learned_addr learned my addr 172.21.15.146:0/374555103 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f581affd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58140e5b60 0x7f5814038560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.156+0000 7f581bfff640 1 --2- 172.21.15.146:0/374555103 >> v2:172.21.15.146:3300/0 conn(0x7f5814038aa0 0x7f5814050780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7ff0bd1af640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0a4003a70 0x7ff0a4003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7ff0bc9ae640 1 -- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a40051a0 msgr2=0x7ff0a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a40051a0 0x7ff0a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7ff0bc9ae640 1 -- 172.21.15.146:0/994225125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0a4003a70 msgr2=0x7ff0a4003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0a4003a70 0x7ff0a4003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7ff0bc9ae640 1 -- 172.21.15.146:0/994225125 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a4003520 con 0x7ff0a4004800 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7ff0bd9b0640 1 -- 172.21.15.146:0/994225125 reap_dead start 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f5b5097f640 1 -- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c004800 msgr2=0x7f5b3c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f5b5097f640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c004800 0x7f5b3c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f5b5097f640 1 -- 172.21.15.146:0/476944920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b3c005150 msgr2=0x7f5b3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f5b5097f640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b3c005150 0x7f5b3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f5b5097f640 1 -- 172.21.15.146:0/476944920 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c003520 con 0x7f5b3c0039b0 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f5b51180640 1 -- 172.21.15.146:0/476944920 reap_dead start 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f660bfff640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65f4003b40 0x7f65f4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f660bfff640 1 -- 172.21.15.146:0/3453052840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4005150 msgr2=0x7f65f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f660bfff640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4005150 0x7f65f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f660bfff640 1 -- 172.21.15.146:0/3453052840 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c0d4470 con 0x7f65f4004800 2024-04-25T15:20:11.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f6610f32640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65f4003b40 0x7f65f4003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f581affd640 1 -- 172.21.15.146:0/374555103 >> v2:172.21.15.177:3300/0 conn(0x7f581404ec80 msgr2=0x7f58140060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f581affd640 1 --2- 172.21.15.146:0/374555103 >> v2:172.21.15.177:3300/0 conn(0x7f581404ec80 0x7f58140060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f581affd640 1 -- 172.21.15.146:0/374555103 >> v2:172.21.15.146:3300/0 conn(0x7f5814038aa0 msgr2=0x7f5814050780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f581affd640 1 --2- 172.21.15.146:0/374555103 >> v2:172.21.15.146:3300/0 conn(0x7f5814038aa0 0x7f5814050780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.157+0000 7f581affd640 1 -- 172.21.15.146:0/374555103 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581403aa40 con 0x7f58140e5b60 2024-04-25T15:20:11.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f8e94c35640 1 -- 172.21.15.146:0/180515553 wait complete. 2024-04-25T15:20:11.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f660bfff640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65f4004800 0x7f65f4004c10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f660c0b82e0 tx=0x7f65f4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e983fbd40b7991 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdab0f10640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdaac06c550 0x7fdaac0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f5b5097f640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b3c0039b0 0x7f5b3c003de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009a90 tx=0x7f5b3c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88d6cf417a28c557 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f1682a97640 1 Processor -- start 2024-04-25T15:20:11.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0a4004800 0x7ff0a4004c10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4004780 tx=0x7ff0a4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be46b86530d00e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdab0f10640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdaac0d00b0 0x7fdaac0d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fc4215d5640 1 Processor -- start 2024-04-25T15:20:11.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660402e070 con 0x7f65f4004800 2024-04-25T15:20:11.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f1682a97640 1 -- start start 2024-04-25T15:20:11.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdab0f10640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaac16d6c0 con 0x7fdaac170530 2024-04-25T15:20:11.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdab0f10640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdaac0d0a00 con 0x7fdaac06c550 2024-04-25T15:20:11.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdab0f10640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fdaac16d9a0 con 0x7fdaac0d00b0 2024-04-25T15:20:11.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6604032600 con 0x7f65f4004800 2024-04-25T15:20:11.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f660b7fe640 1 -- 172.21.15.146:0/3453052840 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604002090 con 0x7f65f4004800 2024-04-25T15:20:11.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7f581affd640 1 --2- 172.21.15.146:0/374555103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58140e5b60 0x7f5814038560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f581400d700 tx=0x7f580802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6736c0585a42538d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdaaaffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdaac0d00b0 0x7fdaac0d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdaab7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdaac06c550 0x7fdaac0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdaaaffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdaac0d00b0 0x7fdaac0d04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45286/0 (socket says 172.21.15.146:45286) 2024-04-25T15:20:11.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fdaaaffd640 1 -- 172.21.15.146:0/3822730133 learned_addr learned my addr 172.21.15.146:0/3822730133 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f581a7fc640 1 -- 172.21.15.146:0/374555103 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808037070 con 0x7f58140e5b60 2024-04-25T15:20:11.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0033070 con 0x7ff0a4004800 2024-04-25T15:20:11.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38036070 con 0x7f5b3c0039b0 2024-04-25T15:20:11.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b002f650 con 0x7ff0a4004800 2024-04-25T15:20:11.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b38031400 con 0x7f5b3c0039b0 2024-04-25T15:20:11.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f5b4b7fe640 1 -- 172.21.15.146:0/476944920 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38009690 con 0x7f5b3c0039b0 2024-04-25T15:20:11.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7ff0affff640 1 -- 172.21.15.146:0/994225125 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0009de0 con 0x7ff0a4004800 2024-04-25T15:20:11.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f6611f34640 1 -- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65f4004800 msgr2=0x7f65f4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f6611f34640 1 --2- 172.21.15.146:0/3453052840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65f4004800 0x7f65f4004c10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f660c0b82e0 tx=0x7f65f4005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f6611733640 1 -- 172.21.15.146:0/3453052840 reap_dead start 2024-04-25T15:20:11.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f581a7fc640 1 -- 172.21.15.146:0/374555103 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580802add0 con 0x7f58140e5b60 2024-04-25T15:20:11.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7f581a7fc640 1 -- 172.21.15.146:0/374555103 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808032660 con 0x7f58140e5b60 2024-04-25T15:20:11.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 -- 172.21.15.146:0/374555103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58140e5b60 msgr2=0x7f5814038560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 -- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0a4004800 msgr2=0x7ff0a4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0a4004800 0x7ff0a4004c10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4004780 tx=0x7ff0a4005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 --2- 172.21.15.146:0/374555103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58140e5b60 0x7f5814038560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f581400d700 tx=0x7f580802a080 comp rx=0 tx=0).stop 2024-04-25T15:20:11.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7fdaab7fe640 1 -- 172.21.15.146:0/3822730133 >> v2:172.21.15.177:3300/0 conn(0x7fdaac0d00b0 msgr2=0x7fdaac0d04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7fdaab7fe640 1 --2- 172.21.15.146:0/3822730133 >> v2:172.21.15.177:3300/0 conn(0x7fdaac0d00b0 0x7fdaac0d04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7fdaab7fe640 1 -- 172.21.15.146:0/3822730133 >> v1:172.21.15.177:6789/0 conn(0x7fdaac170530 legacy=0x7fdaac170940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:20:11.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.159+0000 7fdaab7fe640 1 -- 172.21.15.146:0/3822730133 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac16dc80 con 0x7fdaac06c550 2024-04-25T15:20:11.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7fdaab7fe640 1 --2- 172.21.15.146:0/3822730133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac06c550 0x7fdaac0709d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009a00 tx=0x7fdaa002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d07422bfaf800ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 -- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b3c0039b0 msgr2=0x7f5b3c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b3c0039b0 0x7f5b3c003de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009a90 tx=0x7f5b3c005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 -- 172.21.15.146:0/994225125 shutdown_connections 2024-04-25T15:20:11.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 --2- 172.21.15.146:0/994225125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0a4004800 0x7ff0a4004c10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 -- 172.21.15.146:0/994225125 >> 172.21.15.146:0/994225125 conn(0x7ff0b809b2b0 msgr2=0x7ff0b80b6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 -- 172.21.15.146:0/374555103 shutdown_connections 2024-04-25T15:20:11.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f6611f34640 1 -- 172.21.15.146:0/3453052840 shutdown_connections 2024-04-25T15:20:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 -- 172.21.15.146:0/994225125 shutdown_connections 2024-04-25T15:20:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f6611f34640 1 -- 172.21.15.146:0/3453052840 >> 172.21.15.146:0/3453052840 conn(0x7f660c0b7370 msgr2=0x7f660c0b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 --2- 172.21.15.146:0/374555103 >> v2:172.21.15.146:3300/0 conn(0x7f5814038aa0 0x7f5814050780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 --2- 172.21.15.146:0/374555103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58140e5b60 0x7f5814038560 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 --2- 172.21.15.146:0/374555103 >> v2:172.21.15.177:3300/0 conn(0x7f581404ec80 0x7f58140060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 -- 172.21.15.146:0/476944920 shutdown_connections 2024-04-25T15:20:11.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 --2- 172.21.15.146:0/476944920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b3c0039b0 0x7f5b3c003de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 -- 172.21.15.146:0/476944920 >> 172.21.15.146:0/476944920 conn(0x7f5b44055140 msgr2=0x7f5b44055550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 -- 172.21.15.146:0/374555103 >> 172.21.15.146:0/374555103 conn(0x7f5814054d20 msgr2=0x7f5814055130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 -- 172.21.15.146:0/476944920 shutdown_connections 2024-04-25T15:20:11.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 -- 172.21.15.146:0/374555103 shutdown_connections 2024-04-25T15:20:11.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f58220d8640 1 -- 172.21.15.146:0/374555103 wait complete. 2024-04-25T15:20:11.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7f5b52be3640 1 -- 172.21.15.146:0/476944920 wait complete. 2024-04-25T15:20:11.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.158+0000 7fc4215d5640 1 -- start start 2024-04-25T15:20:11.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f8e94c35640 1 Processor -- start 2024-04-25T15:20:11.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.160+0000 7ff0bf413640 1 -- 172.21.15.146:0/994225125 wait complete. 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f1682a97640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f167403c8b0 0x7f16740514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f1682a97640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1674054b10 con 0x7f1674051a30 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f1682a97640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16740521a0 con 0x7f16740e3a50 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f1682a97640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f1674054e10 con 0x7f167403c8b0 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f1680833640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f16740e3a50 0x7f1674039d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38918/0 (socket says 172.21.15.146:38918) 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f1680833640 1 -- 172.21.15.146:0/2985611845 learned_addr learned my addr 172.21.15.146:0/2985611845 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f8e94c35640 1 -- start start 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc4215d5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc414057bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc4215d5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc414058b90 0x7fc4140380e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc4215d5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc414038620 con 0x7fc4140073d0 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc4215d5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc414057730 con 0x7fc414058b90 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc4215d5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc414056e00 con 0x7fc4140e3a90 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc414057bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc414057bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39612/0 (socket says 172.21.15.146:39612) 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 -- 172.21.15.146:0/872297664 learned_addr learned my addr 172.21.15.146:0/872297664 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 -- 172.21.15.146:0/872297664 >> v2:172.21.15.146:3300/0 conn(0x7fc414058b90 msgr2=0x7fc4140380e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f167bfff640 1 --2- 172.21.15.146:0/2985611845 >> v2:172.21.15.177:3300/0 conn(0x7f167403c8b0 0x7f16740514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fdaaa7fc640 1 -- 172.21.15.146:0/3822730133 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0036070 con 0x7fdaac06c550 2024-04-25T15:20:11.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fdaaa7fc640 1 -- 172.21.15.146:0/3822730133 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa0031600 con 0x7fdaac06c550 2024-04-25T15:20:11.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fdaaa7fc640 1 -- 172.21.15.146:0/3822730133 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa00319b0 con 0x7fdaac06c550 2024-04-25T15:20:11.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292505861 0 0) 0x7f16740521a0 con 0x7f16740e3a50 2024-04-25T15:20:11.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f166c003520 con 0x7f16740e3a50 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1721367553 0 0) 0x7f166c003520 con 0x7f16740e3a50 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 >> v2:172.21.15.177:3300/0 conn(0x7f167403c8b0 msgr2=0x7f16740514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 --2- 172.21.15.146:0/2985611845 >> v2:172.21.15.177:3300/0 conn(0x7f167403c8b0 0x7f16740514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 >> v1:172.21.15.146:6789/0 conn(0x7f1674051a30 legacy=0x7f16740552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1674052480 con 0x7f16740e3a50 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f167c076290 con 0x7f16740e3a50 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 >> v1:172.21.15.177:6789/0 conn(0x7f16740e3a50 legacy=0x7f1674039d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f166c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c004800 0x7f166c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c005150 0x7f166c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1674054e10 con 0x7f166c004800 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f166c00c0a0 con 0x7f166c005150 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f166c00c2a0 con 0x7f166c0039b0 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1680833640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f166c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1681034640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c005150 0x7f166c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41a7fc640 1 --2- 172.21.15.146:0/872297664 >> v2:172.21.15.146:3300/0 conn(0x7fc414058b90 0x7fc4140380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 --2- 172.21.15.146:0/872297664 >> v2:172.21.15.146:3300/0 conn(0x7fc414058b90 0x7fc4140380e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 -- 172.21.15.146:0/872297664 >> v1:172.21.15.177:6789/0 conn(0x7fc4140073d0 legacy=0x7fc4140077e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:20:11.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.161+0000 7fc41affd640 1 -- 172.21.15.146:0/872297664 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4140570e0 con 0x7fc4140e3a90 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1680833640 1 -- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c005150 msgr2=0x7f166c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1680833640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c005150 0x7f166c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1680833640 1 -- 172.21.15.146:0/2985611845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c004800 msgr2=0x7f166c004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1680833640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c004800 0x7f166c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.162+0000 7f1680833640 1 -- 172.21.15.146:0/2985611845 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1674052480 con 0x7f166c0039b0 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f1681034640 1 -- 172.21.15.146:0/2985611845 reap_dead start 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 -- 172.21.15.146:0/3822730133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac06c550 msgr2=0x7fdaac0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 --2- 172.21.15.146:0/3822730133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac06c550 0x7fdaac0709d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009a00 tx=0x7fdaa002a420 comp rx=0 tx=0).stop 2024-04-25T15:20:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fc41affd640 1 --2- 172.21.15.146:0/872297664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc414057bc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009870 tx=0x7fc40c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14e593bb04f5bc24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fc41a7fc640 1 --2- 172.21.15.146:0/872297664 >> v2:172.21.15.146:3300/0 conn(0x7fc414058b90 0x7fc4140380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f58220d8640 1 Processor -- start 2024-04-25T15:20:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f6611f34640 1 -- 172.21.15.146:0/3453052840 shutdown_connections 2024-04-25T15:20:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f6611f34640 1 -- 172.21.15.146:0/3453052840 wait complete. 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7ff0bf413640 1 Processor -- start 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7ff0bf413640 1 -- start start 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 -- 172.21.15.146:0/3822730133 shutdown_connections 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 --2- 172.21.15.146:0/3822730133 >> v2:172.21.15.177:3300/0 conn(0x7fdaac0d00b0 0x7fdaac0d04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 --2- 172.21.15.146:0/3822730133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac06c550 0x7fdaac0709d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 -- 172.21.15.146:0/3822730133 >> 172.21.15.146:0/3822730133 conn(0x7fdaac0cde40 msgr2=0x7fdaac0ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f5b52be3640 1 Processor -- start 2024-04-25T15:20:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f8e94c35640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e880f66c0 0x7f8e880f83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f58220d8640 1 -- start start 2024-04-25T15:20:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 -- 172.21.15.146:0/3822730133 shutdown_connections 2024-04-25T15:20:11.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f8e94c35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880f8930 0x7f8e8804bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f5b52be3640 1 -- start start 2024-04-25T15:20:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e880f66c0 0x7f8e880f83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e880f66c0 0x7f8e880f83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39632/0 (socket says 172.21.15.146:39632) 2024-04-25T15:20:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e8f7fe640 1 -- 172.21.15.146:0/2163171206 learned_addr learned my addr 172.21.15.146:0/2163171206 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e94c35640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e8804c510 0x7f8e88038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e8effd640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880f8930 0x7f8e8804bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7f1680833640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f166c003e20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f167c000b60 tx=0x7f166c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=448ce1bbe83d7f9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c06a0e0 con 0x7f166c0039b0 2024-04-25T15:20:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f167c06e390 con 0x7f166c0039b0 2024-04-25T15:20:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f167b7fe640 1 -- 172.21.15.146:0/2985611845 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c06c790 con 0x7f166c0039b0 2024-04-25T15:20:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e8804c510 0x7f8e88038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8804fe80 con 0x7f8e880f8930 2024-04-25T15:20:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fc419ffb640 1 -- 172.21.15.146:0/872297664 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c037070 con 0x7fc4140e3a90 2024-04-25T15:20:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88050850 con 0x7f8e8804c510 2024-04-25T15:20:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8e88050000 con 0x7f8e880f66c0 2024-04-25T15:20:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fc419ffb640 1 -- 172.21.15.146:0/872297664 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc40c032440 con 0x7fc4140e3a90 2024-04-25T15:20:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fc419ffb640 1 -- 172.21.15.146:0/872297664 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0327f0 con 0x7fc4140e3a90 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 -- 172.21.15.146:0/872297664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 msgr2=0x7fc414057bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 --2- 172.21.15.146:0/872297664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc414057bc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009870 tx=0x7fc40c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 -- 172.21.15.146:0/872297664 shutdown_connections 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 --2- 172.21.15.146:0/872297664 >> v2:172.21.15.146:3300/0 conn(0x7fc414058b90 0x7fc4140380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 --2- 172.21.15.146:0/872297664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc414057bc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 -- 172.21.15.146:0/872297664 >> 172.21.15.146:0/872297664 conn(0x7fc41403abf0 msgr2=0x7fc41403b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.164+0000 7fc4215d5640 1 -- 172.21.15.146:0/872297664 shutdown_connections 2024-04-25T15:20:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.163+0000 7fdab0f10640 1 -- 172.21.15.146:0/3822730133 wait complete. 2024-04-25T15:20:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f58220d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5814038aa0 0x7f581404e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f58220d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f581404ec80 0x7f58140d2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f58220d8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58140e5b60 0x7f5814181880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f58220d8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f58140e5f70 con 0x7f58140e5b60 2024-04-25T15:20:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7ff0bf413640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a4004800 0x7ff0b80cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f58220d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5814050260 con 0x7f581404ec80 2024-04-25T15:20:11.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f58220d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f58140e60f0 con 0x7f5814038aa0 2024-04-25T15:20:11.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bf413640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b81f8b90 0x7ff0b80b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f581b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5814038aa0 0x7f581404e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bf413640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0b815b9b0 0x7ff0b815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f581b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5814038aa0 0x7f581404e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39648/0 (socket says 172.21.15.146:39648) 2024-04-25T15:20:11.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bf413640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b806c960 con 0x7ff0b81f8b90 2024-04-25T15:20:11.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f581b7fe640 1 -- 172.21.15.146:0/4183360484 learned_addr learned my addr 172.21.15.146:0/4183360484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f581bfff640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58140e5b60 0x7f5814181880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bd1af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a4004800 0x7ff0b80cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bd9b0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0b815b9b0 0x7ff0b815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bd9b0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0b815b9b0 0x7ff0b815bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39654/0 (socket says 172.21.15.146:39654) 2024-04-25T15:20:11.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bd9b0640 1 -- 172.21.15.146:0/33103890 learned_addr learned my addr 172.21.15.146:0/33103890 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80b1ab0 con 0x7ff0a4004800 2024-04-25T15:20:11.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f1682a97640 1 -- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 msgr2=0x7f166c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f1682a97640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f166c003e20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f167c000b60 tx=0x7f166c005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b806cae0 con 0x7ff0b815b9b0 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7fc4215d5640 1 -- 172.21.15.146:0/872297664 wait complete. 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f8e8f7fe640 1 -- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e8804c510 msgr2=0x7f8e88038340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f8e8f7fe640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e8804c510 0x7f8e88038340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f8e8f7fe640 1 -- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880f8930 msgr2=0x7f8e8804bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f8e8f7fe640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880f8930 0x7f8e8804bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.165+0000 7f8e8f7fe640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e880f8d40 con 0x7f8e880f66c0 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f8e8f7fe640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e880f66c0 0x7f8e880f83f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e90000b60 tx=0x7f8e9007bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9007bd30 con 0x7f8e880f66c0 2024-04-25T15:20:11.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e9006d6d0 con 0x7f8e880f66c0 2024-04-25T15:20:11.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7fdab0f10640 1 Processor -- start 2024-04-25T15:20:11.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f581affd640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f581404ec80 0x7f58140d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9006dac0 con 0x7f8e880f66c0 2024-04-25T15:20:11.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b81f8b90 0x7ff0b80b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8804cd40 con 0x7f8e880f66c0 2024-04-25T15:20:11.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8804d300 con 0x7f8e880f66c0 2024-04-25T15:20:11.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.166+0000 7fdab0f10640 1 -- start start 2024-04-25T15:20:11.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f5b52be3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c0039b0 0x7f5b440d8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bc9ae640 1 -- 172.21.15.146:0/33103890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0b815b9b0 msgr2=0x7ff0b815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0b815b9b0 0x7ff0b815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bc9ae640 1 -- 172.21.15.146:0/33103890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a4004800 msgr2=0x7ff0b80cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0a4004800 0x7ff0b80cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f5b52be3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b441821d0 0x7f5b44181550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bc9ae640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b80af3e0 con 0x7ff0b81f8b90 2024-04-25T15:20:11.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f6611f34640 1 Processor -- start 2024-04-25T15:20:11.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f5b52be3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b44181a90 0x7f5b4404c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f6611f34640 1 -- start start 2024-04-25T15:20:11.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bd9b0640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0b815b9b0 0x7ff0b815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:11.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7ff0bc9ae640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b81f8b90 0x7ff0b80b0480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0032000 tx=0x7ff0b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f581b7fe640 1 -- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f581404ec80 msgr2=0x7f58140d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b441821d0 0x7f5b44181550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f5b4bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b441821d0 0x7f5b44181550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39658/0 (socket says 172.21.15.146:39658) 2024-04-25T15:20:11.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f1682a97640 1 -- 172.21.15.146:0/2985611845 shutdown_connections 2024-04-25T15:20:11.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f1682a97640 1 --2- 172.21.15.146:0/2985611845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f166c003e20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f1682a97640 1 -- 172.21.15.146:0/2985611845 >> 172.21.15.146:0/2985611845 conn(0x7f167400c420 msgr2=0x7f167400c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f581b7fe640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f581404ec80 0x7f58140d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b4bfff640 1 -- 172.21.15.146:0/1337365969 learned_addr learned my addr 172.21.15.146:0/1337365969 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44050800 con 0x7f5b44181a90 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b5097f640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c0039b0 0x7f5b440d8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b440eb360 con 0x7f5b3c0039b0 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5b44050980 con 0x7f5b441821d0 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b51180640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b44181a90 0x7f5b4404c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7fc4215d5640 1 Processor -- start 2024-04-25T15:20:11.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7fc4215d5640 1 -- start start 2024-04-25T15:20:11.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.167+0000 7f581b7fe640 1 -- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58140e5b60 msgr2=0x7f5814181880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f581b7fe640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58140e5b60 0x7f5814181880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f581b7fe640 1 -- 172.21.15.146:0/4183360484 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58140e2a80 con 0x7f5814038aa0 2024-04-25T15:20:11.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f581bfff640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58140e5b60 0x7f5814181880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f581b7fe640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5814038aa0 0x7f581404e1c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f581000e870 tx=0x7f581000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0046070 con 0x7ff0b81f8b90 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b002f450 con 0x7ff0b81f8b90 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0033040 con 0x7ff0b81f8b90 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b4bfff640 1 -- 172.21.15.146:0/1337365969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c0039b0 msgr2=0x7f5b440d8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b4bfff640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b3c0039b0 0x7f5b440d8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b4bfff640 1 -- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b44181a90 msgr2=0x7f5b4404c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b4bfff640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b44181a90 0x7f5b4404c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5b4bfff640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b44181ea0 con 0x7f5b441821d0 2024-04-25T15:20:11.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810017070 con 0x7f5814038aa0 2024-04-25T15:20:11.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f581000ee60 con 0x7f5814038aa0 2024-04-25T15:20:11.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581404bde0 con 0x7f5814038aa0 2024-04-25T15:20:11.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f581404c3a0 con 0x7f5814038aa0 2024-04-25T15:20:11.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7fdab0f10640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdaac06c550 0x7fdaac155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.168+0000 7f1682a97640 1 -- 172.21.15.146:0/2985611845 shutdown_connections 2024-04-25T15:20:11.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7f1682a97640 1 -- 172.21.15.146:0/2985611845 wait complete. 2024-04-25T15:20:11.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8e9006dc60 con 0x7f8e880f66c0 2024-04-25T15:20:11.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7fdab0f10640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac0d00b0 0x7fdaac073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b80af6c0 con 0x7ff0b81f8b90 2024-04-25T15:20:11.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7fdaab7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdaac06c550 0x7fdaac155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7fdaab7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdaac06c550 0x7fdaac155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39674/0 (socket says 172.21.15.146:39674) 2024-04-25T15:20:11.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b80afc80 con 0x7ff0b81f8b90 2024-04-25T15:20:11.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f8e7bfff640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8e7407cd90 0x7f8e7407f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaab7fe640 1 -- 172.21.15.146:0/1022303982 learned_addr learned my addr 172.21.15.146:0/1022303982 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.169+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581000cb80 con 0x7f5814038aa0 2024-04-25T15:20:11.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaaaffd640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac0d00b0 0x7fdaac073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdab0f10640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdaac170530 0x7fdaac0737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8e90100950 con 0x7f8e880f66c0 2024-04-25T15:20:11.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1711f0 con 0x7fdaac0d00b0 2024-04-25T15:20:11.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac070f10 con 0x7fdaac170530 2024-04-25T15:20:11.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdaac171370 con 0x7fdaac06c550 2024-04-25T15:20:11.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f5b4bfff640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b441821d0 0x7f5b44181550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c06b0f0 tx=0x7f5b4c08dc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58140e2d60 con 0x7f5814038aa0 2024-04-25T15:20:11.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc4215d5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4140073d0 0x7fc414189d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f8e94c35640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f8e88006840 0x7f8e88038880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e8804da50 con 0x7f8e88006840 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc4215d5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc414058b90 0x7fc4140e6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc4215d5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc4140e65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc4215d5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc41403c840 con 0x7fc4140073d0 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc4215d5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc414037d70 con 0x7fc414058b90 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc4215d5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc41403c9c0 con 0x7fc4140e3a90 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc41affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4140073d0 0x7fc414189d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc41a7fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc414058b90 0x7fc4140e6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc4140e65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f8e88006840 0x7f8e88038880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc4140e65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39690/0 (socket says 172.21.15.146:39690) 2024-04-25T15:20:11.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 -- 172.21.15.146:0/3653061925 learned_addr learned my addr 172.21.15.146:0/3653061925 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0b80b41d0 con 0x7ff0b81f8b90 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaaaffd640 1 -- 172.21.15.146:0/1022303982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdaac06c550 msgr2=0x7fdaac155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaaaffd640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdaac06c550 0x7fdaac155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaaaffd640 1 -- 172.21.15.146:0/1022303982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdaac170530 msgr2=0x7fdaac0737b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaaaffd640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdaac170530 0x7fdaac0737b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaaaffd640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac08a120 con 0x7fdaac0d00b0 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7fdaab7fe640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdaac06c550 0x7fdaac155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fdaaaffd640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac0d00b0 0x7fdaac073270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00e870 tx=0x7fda9c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c08de10 con 0x7f5b441821d0 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4c06d6d0 con 0x7f5b441821d0 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4417de50 con 0x7f5b441821d0 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b44172550 con 0x7f5b441821d0 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.170+0000 7f6611f34640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4004800 0x7f660c06c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6610f32640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4004800 0x7f660c06c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6610f32640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4004800 0x7f660c06c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59988/0 (socket says 172.21.15.146:59988) 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6610f32640 1 -- 172.21.15.146:0/2605893838 learned_addr learned my addr 172.21.15.146:0/2605893838 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6611f34640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c154c90 0x7f660c06ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6611f34640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f660c15c0e0 0x7f660c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0d0f00 con 0x7f65f4004800 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0d4a30 con 0x7f660c15c0e0 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0d1080 con 0x7f660c154c90 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f660bfff640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c154c90 0x7f660c06ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 -- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc414058b90 msgr2=0x7fc4140e6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc414058b90 0x7fc4140e6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 -- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4140073d0 msgr2=0x7fc414189d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4140073d0 0x7fc414189d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 -- 172.21.15.146:0/3653061925 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41404fae0 con 0x7fc4140e3a90 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7fc41b7fe640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc4140e65b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc41c02e0e0 tx=0x7fc41c0754c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f8e8effd640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8e7407cd90 0x7f8e7407f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f8e8ffff640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f8e88006840 0x7f8e88038880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f581001a070 con 0x7f5814038aa0 2024-04-25T15:20:11.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c017070 con 0x7fdaac0d00b0 2024-04-25T15:20:11.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c00ee60 con 0x7fdaac0d00b0 2024-04-25T15:20:11.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f660bfff640 1 -- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f660c15c0e0 msgr2=0x7f660c15c4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.171+0000 7f660bfff640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f660c15c0e0 0x7f660c15c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7f660bfff640 1 -- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4004800 msgr2=0x7f660c06c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7f660bfff640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65f4004800 0x7f660c06c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7f660bfff640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c15dc30 con 0x7f660c154c90 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaac1568d0 con 0x7fdaac0d00b0 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c017070 con 0x7fdaac0d00b0 2024-04-25T15:20:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaac08a6e0 con 0x7fdaac0d00b0 2024-04-25T15:20:11.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7f5807fff640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5800074580 0x7f5800076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c06ec30 con 0x7fc4140e3a90 2024-04-25T15:20:11.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc41c090070 con 0x7fc4140e3a90 2024-04-25T15:20:11.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c08ca80 con 0x7fc4140e3a90 2024-04-25T15:20:11.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c06da80 con 0x7f5b441821d0 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5810095560 con 0x7f5814038aa0 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f581affd640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5800074580 0x7f5800076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f5807fff640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f580007dd40 0x7f58000801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 --> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5810016ec0 con 0x7f580007dd40 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5810095950 con 0x7f5814038aa0 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f5b36ffd640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b440059c0 con 0x7f5b441821d0 2024-04-25T15:20:11.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f581bfff640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f580007dd40 0x7f58000801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc41404ec60 con 0x7fc4140e3a90 2024-04-25T15:20:11.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc41404f570 con 0x7fc4140e3a90 2024-04-25T15:20:11.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdaac050740 con 0x7fdaac0d00b0 2024-04-25T15:20:11.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f581affd640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5800074580 0x7f5800076a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5808034000 tx=0x7f580802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.172+0000 7f1682a97640 1 Processor -- start 2024-04-25T15:20:11.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.173+0000 7f8e8effd640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8e7407cd90 0x7f8e7407f2a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e880060d0 tx=0x7f8e84001590 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5b4c06dca0 con 0x7f5b441821d0 2024-04-25T15:20:11.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f581bfff640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f580007dd40 0x7f58000801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f1682a97640 1 -- start start 2024-04-25T15:20:11.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== osd.0 v2:172.21.15.146:6816/1105656680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e7c039040 con 0x7f8e88006840 2024-04-25T15:20:11.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f660bfff640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c154c90 0x7f660c06ccf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65f400c430 tx=0x7f6604001650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc41404f180 con 0x7fc4140e3a90 2024-04-25T15:20:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f5b497fa640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5b300745b0 0x7f5b30076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.174+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5b4c1114c0 con 0x7f5b441821d0 2024-04-25T15:20:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604042020 con 0x7f660c154c90 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff0b0049070 con 0x7ff0b81f8b90 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7ff0adffb640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff09c074580 0x7ff09c076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff0b00b6a50 con 0x7ff0b81f8b90 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7ff0adffb640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7ff09c07dd40 0x7ff09c0801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0b0042e50 con 0x7ff09c07dd40 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff0b00b6e40 con 0x7ff0b81f8b90 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7ff0bd9b0640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7ff09c07dd40 0x7ff09c0801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fda9c01a070 con 0x7fdaac0d00b0 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7fda93fff640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda980746b0 0x7fda98076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7fdaab7fe640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda980746b0 0x7fda98076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f5b497fa640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5b3007dcf0 0x7f5b30080180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b4c072e50 con 0x7f5b3007dcf0 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f5b5097f640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5b300745b0 0x7f5b30076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5b4c118050 con 0x7f5b441821d0 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f5b51180640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5b3007dcf0 0x7f5b30080180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7f5b51180640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5b3007dcf0 0x7f5b30080180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c0cbd40 con 0x7f660c154c90 2024-04-25T15:20:11.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c0cb780 con 0x7f660c154c90 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6604004d70 con 0x7f660c154c90 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.175+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== osd.6 v2:172.21.15.177:6800/2938680267 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5810016ec0 con 0x7f580007dd40 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7f1682a97640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f16740dae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.176+0000 7f1682a97640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740db360 0x7f1674037800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1680833640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f16740dae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1680833640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f16740dae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39704/0 (socket says 172.21.15.146:39704) 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1680833640 1 -- 172.21.15.146:0/1195935097 learned_addr learned my addr 172.21.15.146:0/1195935097 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1682a97640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16740db770 0x7f1674037d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1682a97640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f16740514f0 con 0x7f16740db360 2024-04-25T15:20:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1682a97640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1674052760 con 0x7f16740db770 2024-04-25T15:20:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1682a97640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1674051670 con 0x7f166c0039b0 2024-04-25T15:20:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== osd.3 v2:172.21.15.146:6808/2287149109 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b4c072e50 con 0x7f5b3007dcf0 2024-04-25T15:20:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7ff0bd1af640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff09c074580 0x7ff09c076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f167bfff640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740db360 0x7f1674037800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604002090 con 0x7f660c154c90 2024-04-25T15:20:11.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65ec001030 con 0x7f660c154c90 2024-04-25T15:20:11.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.177+0000 7f1681034640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16740db770 0x7f1674037d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7ff0bd9b0640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7ff09c07dd40 0x7ff09c0801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fda9c05b6b0 con 0x7fdaac0d00b0 2024-04-25T15:20:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f167bfff640 1 -- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 msgr2=0x7f16740dae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f167bfff640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166c0039b0 0x7f16740dae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f167bfff640 1 -- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16740db770 msgr2=0x7f1674037d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fda93fff640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7fda9807e9c0 0x7fda98080eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda98081580 con 0x7fda9807e9c0 2024-04-25T15:20:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f167bfff640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16740db770 0x7f1674037d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fda9c05b8b0 con 0x7fdaac0d00b0 2024-04-25T15:20:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f167bfff640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1674036ee0 con 0x7f16740db360 2024-04-25T15:20:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fdaab7fe640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda980746b0 0x7fda98076bc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009560 tx=0x7fdaa0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fdaabfff640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7fda9807e9c0 0x7fda98080eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f167bfff640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740db360 0x7f1674037800 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f16740043e0 tx=0x7f166800d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7f5b5097f640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5b300745b0 0x7f5b30076ac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b38033000 tx=0x7f5b38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668004020 con 0x7f16740db360 2024-04-25T15:20:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7f1682a97640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16740371c0 con 0x7f16740db360 2024-04-25T15:20:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7f1682a97640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167400a5d0 con 0x7f16740db360 2024-04-25T15:20:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7ff0bd1af640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff09c074580 0x7ff09c076a90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4006fd0 tx=0x7ff0b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc41c08cc20 con 0x7fc4140e3a90 2024-04-25T15:20:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7fdaabfff640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7fda9807e9c0 0x7fda98080eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7fc4037fe640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc408074580 0x7fc408076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc41c111cc0 con 0x7fc4140e3a90 2024-04-25T15:20:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7fc4037fe640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7fc40807dd40 0x7fc4080801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 --> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc41c099e50 con 0x7fc40807dd40 2024-04-25T15:20:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc41c1120b0 con 0x7fc4140e3a90 2024-04-25T15:20:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== osd.1 v2:172.21.15.146:6824/3545155054 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0b0042e50 con 0x7ff09c07dd40 2024-04-25T15:20:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.178+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6604045020 con 0x7f660c154c90 2024-04-25T15:20:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7f66097fa640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65f80745b0 0x7f65f8076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.179+0000 7f6610f32640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65f80745b0 0x7f65f8076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f66040b62d0 con 0x7f660c154c90 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f66097fa640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f65f807dd80 0x7f65f8080210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f6610f32640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65f80745b0 0x7f65f8076ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6600004650 tx=0x7f6600008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f660402ce50 con 0x7f65f807dd80 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66040bd050 con 0x7f660c154c90 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16680041c0 con 0x7f16740db360 2024-04-25T15:20:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668004020 con 0x7f16740db360 2024-04-25T15:20:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7fc41affd640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc408074580 0x7fc408076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== osd.2 v2:172.21.15.146:6800/3921051373 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda98081580 con 0x7fda9807e9c0 2024-04-25T15:20:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.181+0000 7fc41affd640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc408074580 0x7fc408076a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc40c034000 tx=0x7fc40c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.181+0000 7fc41a7fc640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7fc40807dd40 0x7fc4080801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.180+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16740055a0 con 0x7f16740db360 2024-04-25T15:20:11.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.182+0000 7f6611733640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f65f807dd80 0x7f65f8080210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.182+0000 7fc41a7fc640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7fc40807dd40 0x7fc4080801d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.182+0000 7f6611733640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f65f807dd80 0x7f65f8080210 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.183+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== osd.5 v2:172.21.15.177:6808/2540256124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc41c099e50 con 0x7fc40807dd40 2024-04-25T15:20:11.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.183+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f166801a020 con 0x7f16740db360 2024-04-25T15:20:11.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.184+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== osd.4 v2:172.21.15.177:6817/761443217 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f660402ce50 con 0x7f65f807dd80 2024-04-25T15:20:11.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.184+0000 7f16797fa640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16600748b0 0x7f1660076dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.184+0000 7f1680833640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16600748b0 0x7f1660076dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.185+0000 7f1680833640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16600748b0 0x7f1660076dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f167c06aed0 tx=0x7f167c070030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.185+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1668018020 con 0x7f16740db360 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.185+0000 7f16797fa640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7f166007eb00 0x7f1660080ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.185+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16600816c0 con 0x7f166007eb00 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.186+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f166805ba60 con 0x7f16740db360 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.186+0000 7f1681034640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7f166007eb00 0x7f1660080ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.187+0000 7f1681034640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7f166007eb00 0x7f1660080ff0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.188+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== osd.7 v2:172.21.15.177:6816/1513283347 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16600816c0 con 0x7f166007eb00 2024-04-25T15:20:11.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.204+0000 7f8e94c35640 1 -- 172.21.15.146:0/2163171206 --> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e880485c0 con 0x7f8e88006840 2024-04-25T15:20:11.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.205+0000 7f8e7bfff640 1 -- 172.21.15.146:0/2163171206 <== osd.0 v2:172.21.15.146:6816/1105656680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8e880485c0 con 0x7f8e88006840 2024-04-25T15:20:11.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.205+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f8e88006840 msgr2=0x7f8e88038880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.205+0000 7f8e79ffb640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f8e88006840 0x7f8e88038880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.206+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8e7407cd90 msgr2=0x7f8e7407f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.206+0000 7f8e79ffb640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8e7407cd90 0x7f8e7407f2a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e880060d0 tx=0x7f8e84001590 comp rx=0 tx=0).stop 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.206+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e880f66c0 msgr2=0x7f8e880f83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.206+0000 7f8e79ffb640 1 --2- 172.21.15.146:0/2163171206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e880f66c0 0x7f8e880f83f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8e90000b60 tx=0x7f8e9007bc00 comp rx=0 tx=0).stop 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.206+0000 7f8e8ffff640 1 -- 172.21.15.146:0/2163171206 reap_dead start 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.207+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 shutdown_connections 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.207+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 >> 172.21.15.146:0/2163171206 conn(0x7f8e88054cf0 msgr2=0x7f8e880fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.206+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 --> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f58140055a0 con 0x7f580007dd40 2024-04-25T15:20:11.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.207+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 shutdown_connections 2024-04-25T15:20:11.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.207+0000 7f8e79ffb640 1 -- 172.21.15.146:0/2163171206 wait complete. 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.207+0000 7f5807fff640 1 -- 172.21.15.146:0/4183360484 <== osd.6 v2:172.21.15.177:6800/2938680267 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f58140055a0 con 0x7f580007dd40 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f580007dd40 msgr2=0x7f58000801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f58220d8640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f580007dd40 0x7f58000801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5800074580 msgr2=0x7f5800076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f58220d8640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5800074580 0x7f5800076a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5808034000 tx=0x7f580802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5814038aa0 msgr2=0x7f581404e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f58220d8640 1 --2- 172.21.15.146:0/4183360484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5814038aa0 0x7f581404e1c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f581000e870 tx=0x7f581000ed30 comp rx=0 tx=0).stop 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.208+0000 7f581bfff640 1 -- 172.21.15.146:0/4183360484 reap_dead start 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.209+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 shutdown_connections 2024-04-25T15:20:11.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.209+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 >> 172.21.15.146:0/4183360484 conn(0x7f5814054d20 msgr2=0x7f5814007030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.209+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 shutdown_connections 2024-04-25T15:20:11.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.209+0000 7f58220d8640 1 -- 172.21.15.146:0/4183360484 wait complete. 2024-04-25T15:20:11.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.210+0000 7f5b36ffd640 1 -- 172.21.15.146:0/1337365969 --> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b4404fdf0 con 0x7f5b3007dcf0 2024-04-25T15:20:11.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.211+0000 7fdab0f10640 1 -- 172.21.15.146:0/1022303982 --> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdaac06fb70 con 0x7fda9807e9c0 2024-04-25T15:20:11.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.211+0000 7fda93fff640 1 -- 172.21.15.146:0/1022303982 <== osd.2 v2:172.21.15.146:6800/3921051373 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdaac06fb70 con 0x7fda9807e9c0 2024-04-25T15:20:11.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.211+0000 7f5b497fa640 1 -- 172.21.15.146:0/1337365969 <== osd.3 v2:172.21.15.146:6808/2287149109 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5b4404fdf0 con 0x7f5b3007dcf0 2024-04-25T15:20:11.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 --> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff0b8050740 con 0x7ff09c07dd40 2024-04-25T15:20:11.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5b3007dcf0 msgr2=0x7f5b30080180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7fda9807e9c0 msgr2=0x7fda98080eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7fda91ffb640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7fda9807e9c0 0x7fda98080eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b52be3640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5b3007dcf0 0x7f5b30080180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5b300745b0 msgr2=0x7f5b30076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b52be3640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5b300745b0 0x7f5b30076ac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b38033000 tx=0x7f5b38008030 comp rx=0 tx=0).stop 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b441821d0 msgr2=0x7f5b44181550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b52be3640 1 --2- 172.21.15.146:0/1337365969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b441821d0 0x7f5b44181550 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c06b0f0 tx=0x7f5b4c08dc20 comp rx=0 tx=0).stop 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7f5b51180640 1 -- 172.21.15.146:0/1337365969 reap_dead start 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0adffb640 1 -- 172.21.15.146:0/33103890 <== osd.1 v2:172.21.15.146:6824/3545155054 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff0b8050740 con 0x7ff09c07dd40 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda980746b0 msgr2=0x7fda98076bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.212+0000 7fda91ffb640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda980746b0 0x7fda98076bc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009560 tx=0x7fdaa0008030 comp rx=0 tx=0).stop 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac0d00b0 msgr2=0x7fdaac073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7fda91ffb640 1 --2- 172.21.15.146:0/1022303982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdaac0d00b0 0x7fdaac073270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00e870 tx=0x7fda9c00ed30 comp rx=0 tx=0).stop 2024-04-25T15:20:11.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 shutdown_connections 2024-04-25T15:20:11.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 >> 172.21.15.146:0/1337365969 conn(0x7f5b44055140 msgr2=0x7f5b44053640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7fdaabfff640 1 -- 172.21.15.146:0/1022303982 reap_dead start 2024-04-25T15:20:11.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 shutdown_connections 2024-04-25T15:20:11.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 shutdown_connections 2024-04-25T15:20:11.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 >> 172.21.15.146:0/1022303982 conn(0x7fdaac0cde40 msgr2=0x7fdaac15b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7ff09c07dd40 msgr2=0x7ff09c0801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0bf413640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7ff09c07dd40 0x7ff09c0801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff09c074580 msgr2=0x7ff09c076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0bf413640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff09c074580 0x7ff09c076a90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4006fd0 tx=0x7ff0b4008040 comp rx=0 tx=0).stop 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b81f8b90 msgr2=0x7ff0b80b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.213+0000 7ff0bf413640 1 --2- 172.21.15.146:0/33103890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b81f8b90 0x7ff0b80b0480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0032000 tx=0x7ff0b0009750 comp rx=0 tx=0).stop 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7f5b52be3640 1 -- 172.21.15.146:0/1337365969 wait complete. 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 shutdown_connections 2024-04-25T15:20:11.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7fda91ffb640 1 -- 172.21.15.146:0/1022303982 wait complete. 2024-04-25T15:20:11.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7ff0bd9b0640 1 -- 172.21.15.146:0/33103890 reap_dead start 2024-04-25T15:20:11.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 shutdown_connections 2024-04-25T15:20:11.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 >> 172.21.15.146:0/33103890 conn(0x7ff0b809b2b0 msgr2=0x7ff0b815f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.214+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 --> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc41404f180 con 0x7fc40807dd40 2024-04-25T15:20:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.215+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 shutdown_connections 2024-04-25T15:20:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.215+0000 7ff0bf413640 1 -- 172.21.15.146:0/33103890 wait complete. 2024-04-25T15:20:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.215+0000 7fc4037fe640 1 -- 172.21.15.146:0/3653061925 <== osd.5 v2:172.21.15.177:6808/2540256124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc41404f180 con 0x7fc40807dd40 2024-04-25T15:20:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.216+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 --> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f65ec002dd0 con 0x7f65f807dd80 2024-04-25T15:20:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.216+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7fc40807dd40 msgr2=0x7fc4080801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.216+0000 7fc4215d5640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7fc40807dd40 0x7fc4080801d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.216+0000 7f66097fa640 1 -- 172.21.15.146:0/2605893838 <== osd.4 v2:172.21.15.177:6817/761443217 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f65ec002dd0 con 0x7f65f807dd80 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.216+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc408074580 msgr2=0x7fc408076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.216+0000 7fc4215d5640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc408074580 0x7fc408076a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc40c034000 tx=0x7fc40c02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 msgr2=0x7fc4140e65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc4215d5640 1 --2- 172.21.15.146:0/3653061925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4140e3a90 0x7fc4140e65b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc41c02e0e0 tx=0x7fc41c0754c0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc41b7fe640 1 -- 172.21.15.146:0/3653061925 reap_dead start 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 shutdown_connections 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 >> 172.21.15.146:0/3653061925 conn(0x7fc41403abf0 msgr2=0x7fc414039bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 shutdown_connections 2024-04-25T15:20:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.217+0000 7fc4215d5640 1 -- 172.21.15.146:0/3653061925 wait complete. 2024-04-25T15:20:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.218+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 --> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1674036c90 con 0x7f166007eb00 2024-04-25T15:20:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.218+0000 7f16797fa640 1 -- 172.21.15.146:0/1195935097 <== osd.7 v2:172.21.15.177:6816/1513283347 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1674036c90 con 0x7f166007eb00 2024-04-25T15:20:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.218+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f65f807dd80 msgr2=0x7f65f8080210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f6611f34640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f65f807dd80 0x7f65f8080210 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65f80745b0 msgr2=0x7f65f8076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f6611f34640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65f80745b0 0x7f65f8076ac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6600004650 tx=0x7f6600008040 comp rx=0 tx=0).stop 2024-04-25T15:20:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c154c90 msgr2=0x7f660c06ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f6611f34640 1 --2- 172.21.15.146:0/2605893838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f660c154c90 0x7f660c06ccf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65f400c430 tx=0x7f6604001650 comp rx=0 tx=0).stop 2024-04-25T15:20:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7f166007eb00 msgr2=0x7f1660080ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f1666ffd640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7f166007eb00 0x7f1660080ff0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.219+0000 7f6611733640 1 -- 172.21.15.146:0/2605893838 reap_dead start 2024-04-25T15:20:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 shutdown_connections 2024-04-25T15:20:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 >> 172.21.15.146:0/2605893838 conn(0x7f660c0b7370 msgr2=0x7f660c161d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16600748b0 msgr2=0x7f1660076dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 shutdown_connections 2024-04-25T15:20:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f6611f34640 1 -- 172.21.15.146:0/2605893838 wait complete. 2024-04-25T15:20:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16600748b0 0x7f1660076dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f167c06aed0 tx=0x7f167c070030 comp rx=0 tx=0).stop 2024-04-25T15:20:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740db360 msgr2=0x7f1674037800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 --2- 172.21.15.146:0/1195935097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740db360 0x7f1674037800 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f16740043e0 tx=0x7f166800d8f0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1681034640 1 -- 172.21.15.146:0/1195935097 reap_dead start 2024-04-25T15:20:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 shutdown_connections 2024-04-25T15:20:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 >> 172.21.15.146:0/1195935097 conn(0x7f167400c420 msgr2=0x7f16740e6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 shutdown_connections 2024-04-25T15:20:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.220+0000 7f1666ffd640 1 -- 172.21.15.146:0/1195935097 wait complete. 2024-04-25T15:20:11.273 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.274 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:20:11.275 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:20:11.276 INFO:teuthology.orchestra.run.smithi146.stdout:55834574859 2024-04-25T15:20:11.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:20:11.277 INFO:teuthology.orchestra.run.smithi146.stdout:55834574859 2024-04-25T15:20:11.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:20:11.278 INFO:teuthology.orchestra.run.smithi146.stdout:55834574857 2024-04-25T15:20:11.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:20:11.279 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:20:11.280 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:20:11.281 INFO:teuthology.orchestra.run.smithi146.stdout:55834574859 2024-04-25T15:20:11.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:20:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.530+0000 7f8d129ce640 1 Processor -- start 2024-04-25T15:20:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.530+0000 7f8d129ce640 1 -- start start 2024-04-25T15:20:11.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.531+0000 7fe84aff8640 1 Processor -- start 2024-04-25T15:20:11.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.531+0000 7fe84aff8640 1 -- start start 2024-04-25T15:20:11.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.531+0000 7f6305822640 1 Processor -- start 2024-04-25T15:20:11.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.531+0000 7f6305822640 1 -- start start 2024-04-25T15:20:11.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.532+0000 7f59812eb640 1 Processor -- start 2024-04-25T15:20:11.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.532+0000 7f8d129ce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d040e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f493c7c7640 1 Processor -- start 2024-04-25T15:20:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d129ce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8d040500b0 0x7f8d04053ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d129ce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d04053030 con 0x7f8d040e47a0 2024-04-25T15:20:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d129ce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8d0404f6c0 con 0x7f8d040500b0 2024-04-25T15:20:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d119cc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d040e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d119cc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d040e12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39706/0 (socket says 172.21.15.146:39706) 2024-04-25T15:20:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d119cc640 1 -- 172.21.15.146:0/2384089572 learned_addr learned my addr 172.21.15.146:0/2384089572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f8d129ce640 1 -- 172.21.15.146:0/2384089572 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8d040532d0 con 0x7f8d04006090 2024-04-25T15:20:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f6305822640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f63000b5dd0 0x7f6300164170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7f493c7c7640 1 -- start start 2024-04-25T15:20:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.532+0000 7f59812eb640 1 -- start start 2024-04-25T15:20:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6305822640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6300167df0 0x7f630006fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7fe84aff8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8440d1270 0x7fe8440b8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6305822640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6300070530 0x7f630006d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6304820640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f63000b5dd0 0x7f6300164170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6304820640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f63000b5dd0 0x7f6300164170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45358/0 (socket says 172.21.15.146:45358) 2024-04-25T15:20:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6304820640 1 -- 172.21.15.146:0/2927340162 learned_addr learned my addr 172.21.15.146:0/2927340162 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f8d121cd640 1 --2- 172.21.15.146:0/2384089572 >> v2:172.21.15.146:3300/0 conn(0x7f8d040500b0 0x7f8d04053ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe64b59e640 1 Processor -- start 2024-04-25T15:20:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.533+0000 7fe84aff8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8440cfb00 con 0x7fe8440d30d0 2024-04-25T15:20:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6305021640 1 --2- 172.21.15.146:0/2927340162 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6300070530 0x7f630006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe84aff8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8440cfde0 con 0x7fe8440b2c50 2024-04-25T15:20:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f63001646b0 con 0x7f6300167df0 2024-04-25T15:20:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f63001685d0 con 0x7f63000b5dd0 2024-04-25T15:20:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f63001688b0 con 0x7f6300070530 2024-04-25T15:20:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe84aff8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe8440d00c0 con 0x7fe8440d1270 2024-04-25T15:20:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe849595640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8440d1270 0x7fe8440b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe849595640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8440d1270 0x7fe8440b8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60016/0 (socket says 172.21.15.146:60016) 2024-04-25T15:20:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe64b59e640 1 -- start start 2024-04-25T15:20:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe849595640 1 -- 172.21.15.146:0/1495961987 learned_addr learned my addr 172.21.15.146:0/1495961987 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875047594 0 0) 0x7fe8440cfde0 con 0x7fe8440b2c50 2024-04-25T15:20:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe834003520 con 0x7fe8440b2c50 2024-04-25T15:20:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fd064b10640 1 Processor -- start 2024-04-25T15:20:11.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f59812eb640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 0x7f597c0aee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.534+0000 7f8d119cc640 1 -- 172.21.15.146:0/2384089572 >> v2:172.21.15.146:3300/0 conn(0x7f8d040500b0 msgr2=0x7f8d04053ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fd064b10640 1 -- start start 2024-04-25T15:20:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1297561509 0 0) 0x7fe834003520 con 0x7fe8440b2c50 2024-04-25T15:20:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f597bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 0x7f597c0aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f62fffff640 1 --2- 172.21.15.146:0/2927340162 >> v2:172.21.15.146:3300/0 conn(0x7f6300167df0 0x7f630006fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f597bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 0x7f597c0aee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39722/0 (socket says 172.21.15.146:39722) 2024-04-25T15:20:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 >> v2:172.21.15.146:3300/0 conn(0x7fe8440d1270 msgr2=0x7fe8440b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f59812eb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f597c0d1f00 0x7f597c0d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f597b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f597c0ce2a0 0x7f597c0ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40580/0 (socket says 172.21.15.146:40580) 2024-04-25T15:20:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f597b7fe640 1 -- 172.21.15.146:0/2459774415 learned_addr learned my addr 172.21.15.146:0/2459774415 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597c0cc540 con 0x7f597c0ce2a0 2024-04-25T15:20:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f597c0cc820 con 0x7f597c0d1f00 2024-04-25T15:20:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f597c0ccb00 con 0x7f597c164410 2024-04-25T15:20:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fe8437fe640 1 --2- 172.21.15.146:0/1495961987 >> v2:172.21.15.146:3300/0 conn(0x7fe8440d1270 0x7fe8440b8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 >> v1:172.21.15.146:6789/0 conn(0x7fe8440d30d0 legacy=0x7fe8440b7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8440d03a0 con 0x7fe8440b2c50 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fe830001670 con 0x7fe8440b2c50 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 >> v1:172.21.15.177:6789/0 conn(0x7fe8440b2c50 legacy=0x7fe8440d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834003a00 0x7fe834003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe834004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe834005270 0x7fe83400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe8440d00c0 con 0x7fe834003a00 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe83400c190 con 0x7fe834005270 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe83400c390 con 0x7fe834004800 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe843fff640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe834004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe848d94640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834003a00 0x7fe834003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe849595640 1 --2- 172.21.15.146:0/1495961987 >> v2:172.21.15.146:3300/0 conn(0x7fe8440d1270 0x7fe8440b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7fe849595640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe834005270 0x7fe83400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f597affd640 1 -- 172.21.15.146:0/2459774415 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872165415 0 0) 0x7f597c0cc540 con 0x7f597c0ce2a0 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f597affd640 1 -- 172.21.15.146:0/2459774415 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5964003520 con 0x7f597c0ce2a0 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f5980aea640 1 --2- 172.21.15.146:0/2459774415 >> v2:172.21.15.146:3300/0 conn(0x7f597c0d1f00 0x7f597c0d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f597bfff640 1 -- 172.21.15.146:0/2459774415 >> v2:172.21.15.146:3300/0 conn(0x7f597c0d1f00 msgr2=0x7f597c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f597bfff640 1 --2- 172.21.15.146:0/2459774415 >> v2:172.21.15.146:3300/0 conn(0x7f597c0d1f00 0x7f597c0d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f597bfff640 1 -- 172.21.15.146:0/2459774415 >> v1:172.21.15.146:6789/0 conn(0x7f597c0ce2a0 legacy=0x7f597c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f8d109ca640 1 -- 172.21.15.146:0/2384089572 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802732219 0 0) 0x7f8d04053030 con 0x7f8d040e47a0 2024-04-25T15:20:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f8d119cc640 1 --2- 172.21.15.146:0/2384089572 >> v2:172.21.15.146:3300/0 conn(0x7f8d040500b0 0x7f8d04053ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f8d119cc640 1 -- 172.21.15.146:0/2384089572 >> v1:172.21.15.177:6789/0 conn(0x7f8d040e47a0 legacy=0x7f8d0404f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f8d119cc640 1 -- 172.21.15.146:0/2384089572 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d040535b0 con 0x7f8d04006090 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f8d121cd640 1 --2- 172.21.15.146:0/2384089572 >> v2:172.21.15.146:3300/0 conn(0x7f8d040500b0 0x7f8d04053ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f62fffff640 1 -- 172.21.15.146:0/2927340162 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6300070530 msgr2=0x7f630006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f62fffff640 1 --2- 172.21.15.146:0/2927340162 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6300070530 0x7f630006d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f62fffff640 1 -- 172.21.15.146:0/2927340162 >> v2:172.21.15.177:3300/0 conn(0x7f63000b5dd0 msgr2=0x7f6300164170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f62fffff640 1 --2- 172.21.15.146:0/2927340162 >> v2:172.21.15.177:3300/0 conn(0x7f63000b5dd0 0x7f6300164170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f62fffff640 1 -- 172.21.15.146:0/2927340162 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6300168b90 con 0x7f6300167df0 2024-04-25T15:20:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f493c7c7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49340b1e30 0x7f49340b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f493c7c7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f49340b72a0 0x7f49340b76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f493c7c7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f493406f290 con 0x7f49340d2490 2024-04-25T15:20:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f493c7c7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f49340cfd30 con 0x7f49340b1e30 2024-04-25T15:20:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.535+0000 7f493c7c7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f49340b7bf0 con 0x7f49340b72a0 2024-04-25T15:20:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f493a563640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49340b1e30 0x7f49340b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f493a563640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49340b1e30 0x7f49340b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60048/0 (socket says 172.21.15.146:60048) 2024-04-25T15:20:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f493a563640 1 -- 172.21.15.146:0/815751572 learned_addr learned my addr 172.21.15.146:0/815751572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f4939d62640 1 --2- 172.21.15.146:0/815751572 >> v2:172.21.15.177:3300/0 conn(0x7f49340b72a0 0x7f49340b76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.536+0000 7f26fab96640 1 Processor -- start 2024-04-25T15:20:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f26fab96640 1 -- start start 2024-04-25T15:20:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f597bfff640 1 -- 172.21.15.146:0/2459774415 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c0ccde0 con 0x7f597c164410 2024-04-25T15:20:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe64b59e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe644048630 0x7fe6440ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe64b59e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644006aa0 0x7fe644006eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe64b59e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe644050240 con 0x7fe644048100 2024-04-25T15:20:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe64b59e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe644050520 con 0x7fe644048630 2024-04-25T15:20:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe64b59e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe644006580 con 0x7fe644006aa0 2024-04-25T15:20:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f8d119cc640 1 --2- 172.21.15.146:0/2384089572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d040e12c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009870 tx=0x7f8cfc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc977b7127f7e403 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe649d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe644048630 0x7fe6440ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe649d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe644048630 0x7fe6440ebdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45386/0 (socket says 172.21.15.146:45386) 2024-04-25T15:20:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f4939d62640 1 -- 172.21.15.146:0/815751572 >> v2:172.21.15.146:3300/0 conn(0x7f49340b1e30 msgr2=0x7f49340b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f4939d62640 1 --2- 172.21.15.146:0/815751572 >> v2:172.21.15.146:3300/0 conn(0x7f49340b1e30 0x7f49340b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f4939d62640 1 -- 172.21.15.146:0/815751572 >> v1:172.21.15.177:6789/0 conn(0x7f49340d2490 legacy=0x7f49340d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7f4939d62640 1 -- 172.21.15.146:0/815751572 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49340b3e40 con 0x7f49340b72a0 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe649d9b640 1 -- 172.21.15.146:0/2288716757 learned_addr learned my addr 172.21.15.146:0/2288716757 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe64ad9d640 1 --2- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644006aa0 0x7fe644006eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd064b10640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd058006920 0x7fd058006d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe843fff640 1 -- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe834005270 msgr2=0x7fe83400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe843fff640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe834005270 0x7fe83400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe843fff640 1 -- 172.21.15.146:0/1495961987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834003a00 msgr2=0x7fe834003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe843fff640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834003a00 0x7fe834003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe843fff640 1 -- 172.21.15.146:0/1495961987 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe834003520 con 0x7fe834004800 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe849595640 1 -- 172.21.15.146:0/1495961987 reap_dead start 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f8d109ca640 1 -- 172.21.15.146:0/2384089572 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc037070 con 0x7f8d04006090 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f8d109ca640 1 -- 172.21.15.146:0/2384089572 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cfc032440 con 0x7f8d04006090 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe649d9b640 1 -- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644006aa0 msgr2=0x7fe644006eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fe649d9b640 1 --2- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644006aa0 0x7fe644006eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fe649d9b640 1 -- 172.21.15.146:0/2288716757 >> v1:172.21.15.146:6789/0 conn(0x7fe644048100 legacy=0x7fe6440eb6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd064b10640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd05804fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd064b10640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd058007270 con 0x7fd0580dfdd0 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd064b10640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd0580e01e0 con 0x7fd058006920 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd064b10640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd05804f470 con 0x7fd05804f900 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd05f7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fd0580dfdd0 0x7fd0580396f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38958/0 (socket says 172.21.15.146:38958) 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.537+0000 7fd05f7fe640 1 -- 172.21.15.146:0/3928127630 learned_addr learned my addr 172.21.15.146:0/3928127630 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05ffff640 1 --2- 172.21.15.146:0/3928127630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd05804fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f8d109ca640 1 -- 172.21.15.146:0/2384089572 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc032810 con 0x7f8d04006090 2024-04-25T15:20:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f62fffff640 1 --2- 172.21.15.146:0/2927340162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f630006fff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63000727a0 tx=0x7f62f002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179745123a12c8b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05effd640 1 --2- 172.21.15.146:0/3928127630 >> v2:172.21.15.146:3300/0 conn(0x7fd058006920 0x7fd058006d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f597bfff640 1 --2- 172.21.15.146:0/2459774415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 0x7f597c0aee00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f59700097a0 tx=0x7f597002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1983f283e53e01fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05ffff640 1 -- 172.21.15.146:0/3928127630 >> v2:172.21.15.146:3300/0 conn(0x7fd058006920 msgr2=0x7fd058006d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05ffff640 1 --2- 172.21.15.146:0/3928127630 >> v2:172.21.15.146:3300/0 conn(0x7fd058006920 0x7fd058006d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05ffff640 1 -- 172.21.15.146:0/3928127630 >> v1:172.21.15.177:6789/0 conn(0x7fd0580dfdd0 legacy=0x7fd0580396f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05e7fc640 1 -- 172.21.15.146:0/3928127630 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527403425 0 0) 0x7fd04c001110 con 0x7fd0580dfdd0 2024-04-25T15:20:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fd05ffff640 1 -- 172.21.15.146:0/3928127630 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05803a3c0 con 0x7fd05804f900 2024-04-25T15:20:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fe649d9b640 1 -- 172.21.15.146:0/2288716757 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6440ec4b0 con 0x7fe644048630 2024-04-25T15:20:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f597affd640 1 -- 172.21.15.146:0/2459774415 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970037070 con 0x7f597c164410 2024-04-25T15:20:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7fe843fff640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe834004c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe838001750 tx=0x7fe834005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=688a503dff0c9f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f62ff7fe640 1 -- 172.21.15.146:0/2927340162 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0037070 con 0x7f6300167df0 2024-04-25T15:20:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f62ff7fe640 1 -- 172.21.15.146:0/2927340162 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f0032440 con 0x7f6300167df0 2024-04-25T15:20:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f62ff7fe640 1 -- 172.21.15.146:0/2927340162 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f002ae60 con 0x7f6300167df0 2024-04-25T15:20:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f597affd640 1 -- 172.21.15.146:0/2459774415 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5970032440 con 0x7f597c164410 2024-04-25T15:20:11.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f597affd640 1 -- 172.21.15.146:0/2459774415 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970037070 con 0x7f597c164410 2024-04-25T15:20:11.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.538+0000 7f4939d62640 1 --2- 172.21.15.146:0/815751572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340b76b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4934074340 tx=0x7f492c02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12160df003822b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f4939561640 1 -- 172.21.15.146:0/815751572 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c007740 con 0x7f49340b72a0 2024-04-25T15:20:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f4939561640 1 -- 172.21.15.146:0/815751572 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f492c0078e0 con 0x7f49340b72a0 2024-04-25T15:20:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f4939561640 1 -- 172.21.15.146:0/815751572 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c007cb0 con 0x7f49340b72a0 2024-04-25T15:20:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f26fab96640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f26ec0547b0 0x7f26ec006d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f8d129ce640 1 -- 172.21.15.146:0/2384089572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 msgr2=0x7f8d040e12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f26fab96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26ec0e3ca0 con 0x7f26ec0e3890 2024-04-25T15:20:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f26fab96640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26ec007990 con 0x7f26ec007290 2024-04-25T15:20:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f26fab96640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f26ec007c70 con 0x7f26ec0547b0 2024-04-25T15:20:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe838001370 con 0x7fe834004800 2024-04-25T15:20:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8380052d0 con 0x7fe834004800 2024-04-25T15:20:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fe8437fe640 1 -- 172.21.15.146:0/1495961987 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83802abc0 con 0x7fe834004800 2024-04-25T15:20:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f8d129ce640 1 --2- 172.21.15.146:0/2384089572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d040e12c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009870 tx=0x7f8cfc02a270 comp rx=0 tx=0).stop 2024-04-25T15:20:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fd05ffff640 1 --2- 172.21.15.146:0/3928127630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd05804fd10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd05400ba40 tx=0x7fd05402c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de65a1524af7b73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f26fa395640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f26ec007290 0x7f26ec03d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38964/0 (socket says 172.21.15.146:38964) 2024-04-25T15:20:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f26fa395640 1 -- 172.21.15.146:0/1166360629 learned_addr learned my addr 172.21.15.146:0/1166360629 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f8d129ce640 1 -- 172.21.15.146:0/2384089572 shutdown_connections 2024-04-25T15:20:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f8d129ce640 1 --2- 172.21.15.146:0/2384089572 >> v2:172.21.15.146:3300/0 conn(0x7f8d040500b0 0x7f8d04053ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f8d129ce640 1 --2- 172.21.15.146:0/2384089572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d040e12c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7f8d129ce640 1 -- 172.21.15.146:0/2384089572 >> 172.21.15.146:0/2384089572 conn(0x7f8d04054cf0 msgr2=0x7f8d04055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fd05e7fc640 1 -- 172.21.15.146:0/3928127630 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054032750 con 0x7fd05804f900 2024-04-25T15:20:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fd05e7fc640 1 -- 172.21.15.146:0/3928127630 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd054030050 con 0x7fd05804f900 2024-04-25T15:20:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.539+0000 7fd05e7fc640 1 -- 172.21.15.146:0/3928127630 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054032b10 con 0x7fd05804f900 2024-04-25T15:20:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f8d129ce640 1 -- 172.21.15.146:0/2384089572 shutdown_connections 2024-04-25T15:20:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338507496 0 0) 0x7f26ec007990 con 0x7f26ec007290 2024-04-25T15:20:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f8d129ce640 1 -- 172.21.15.146:0/2384089572 wait complete. 2024-04-25T15:20:11.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 msgr2=0x7f597c0aee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 msgr2=0x7f630006fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f6305822640 1 --2- 172.21.15.146:0/2927340162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f630006fff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63000727a0 tx=0x7f62f002a110 comp rx=0 tx=0).stop 2024-04-25T15:20:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f9393640 1 --2- 172.21.15.146:0/1166360629 >> v2:172.21.15.177:3300/0 conn(0x7f26ec0547b0 0x7f26ec006d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 --2- 172.21.15.146:0/2459774415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 0x7f597c0aee00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f59700097a0 tx=0x7f597002a180 comp rx=0 tx=0).stop 2024-04-25T15:20:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 shutdown_connections 2024-04-25T15:20:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 shutdown_connections 2024-04-25T15:20:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 --2- 172.21.15.146:0/2459774415 >> v2:172.21.15.146:3300/0 conn(0x7f597c0d1f00 0x7f597c0d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 --2- 172.21.15.146:0/2459774415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c164410 0x7f597c0aee00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 >> 172.21.15.146:0/2459774415 conn(0x7f597c09b2b0 msgr2=0x7f597c0b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 -- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 msgr2=0x7fe834004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe834004c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe838001750 tx=0x7fe834005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e0003520 con 0x7f26ec007290 2024-04-25T15:20:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 -- 172.21.15.146:0/1495961987 shutdown_connections 2024-04-25T15:20:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 shutdown_connections 2024-04-25T15:20:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1495961987 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe834004c10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 -- 172.21.15.146:0/1495961987 >> 172.21.15.146:0/1495961987 conn(0x7fe8440b51b0 msgr2=0x7fe8440b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722052465 0 0) 0x7f26ec0e3ca0 con 0x7f26ec0e3890 2024-04-25T15:20:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f493c7c7640 1 -- 172.21.15.146:0/815751572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 msgr2=0x7f49340b76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f493c7c7640 1 --2- 172.21.15.146:0/815751572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340b76b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4934074340 tx=0x7f492c02a5f0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 -- 172.21.15.146:0/1495961987 shutdown_connections 2024-04-25T15:20:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f59812eb640 1 -- 172.21.15.146:0/2459774415 wait complete. 2024-04-25T15:20:11.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7fe84aff8640 1 -- 172.21.15.146:0/1495961987 wait complete. 2024-04-25T15:20:11.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26ec007990 con 0x7f26ec0e3890 2024-04-25T15:20:11.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f6305822640 1 --2- 172.21.15.146:0/2927340162 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6300070530 0x7f630006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1047998959 0 0) 0x7f26e0003520 con 0x7f26ec007290 2024-04-25T15:20:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 >> v2:172.21.15.177:3300/0 conn(0x7f26ec0547b0 msgr2=0x7f26ec006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 --2- 172.21.15.146:0/1166360629 >> v2:172.21.15.177:3300/0 conn(0x7f26ec0547b0 0x7f26ec006d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f6305822640 1 --2- 172.21.15.146:0/2927340162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f630006fff0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 -- 172.21.15.146:0/3928127630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 msgr2=0x7fd05804fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f6305822640 1 --2- 172.21.15.146:0/2927340162 >> v2:172.21.15.177:3300/0 conn(0x7f63000b5dd0 0x7f6300164170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f493c7c7640 1 -- 172.21.15.146:0/815751572 shutdown_connections 2024-04-25T15:20:11.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f493c7c7640 1 --2- 172.21.15.146:0/815751572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340b76b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 >> 172.21.15.146:0/2927340162 conn(0x7f63000732b0 msgr2=0x7f63000736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f493c7c7640 1 --2- 172.21.15.146:0/815751572 >> v2:172.21.15.146:3300/0 conn(0x7f49340b1e30 0x7f49340b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f493c7c7640 1 -- 172.21.15.146:0/815751572 >> 172.21.15.146:0/815751572 conn(0x7f493409b2b0 msgr2=0x7f493409b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f493c7c7640 1 -- 172.21.15.146:0/815751572 shutdown_connections 2024-04-25T15:20:11.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f493c7c7640 1 -- 172.21.15.146:0/815751572 wait complete. 2024-04-25T15:20:11.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 >> v1:172.21.15.146:6789/0 conn(0x7f26ec0e3890 legacy=0x7f26ec0540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 --2- 172.21.15.146:0/3928127630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd05804fd10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd05400ba40 tx=0x7fd05402c850 comp rx=0 tx=0).stop 2024-04-25T15:20:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec03c950 con 0x7f26ec007290 2024-04-25T15:20:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f26f4076290 con 0x7f26ec007290 2024-04-25T15:20:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fe649d9b640 1 --2- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048630 0x7fe6440ebdb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe64c052040 tx=0x7fe64c0a6eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e76ad78491391b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 -- 172.21.15.146:0/3928127630 shutdown_connections 2024-04-25T15:20:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 --2- 172.21.15.146:0/3928127630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd05804fd10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 --2- 172.21.15.146:0/3928127630 >> v2:172.21.15.146:3300/0 conn(0x7fd058006920 0x7fd058006d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 -- 172.21.15.146:0/3928127630 >> 172.21.15.146:0/3928127630 conn(0x7fd058054cf0 msgr2=0x7fd058055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 shutdown_connections 2024-04-25T15:20:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 >> v1:172.21.15.177:6789/0 conn(0x7f26ec007290 legacy=0x7f26ec03d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f6305822640 1 -- 172.21.15.146:0/2927340162 wait complete. 2024-04-25T15:20:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26e0003b40 0x7f26e0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 -- 172.21.15.146:0/3928127630 shutdown_connections 2024-04-25T15:20:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7fd064b10640 1 -- 172.21.15.146:0/3928127630 wait complete. 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e0004800 0x7f26e0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.540+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec007c70 con 0x7f26e0004800 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f26e000bfb0 con 0x7f26e0003b40 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f26e000c1b0 con 0x7f26e0005150 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26f9b94640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26e0003b40 0x7f26e0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26fa395640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26fa395640 1 -- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26e0003b40 msgr2=0x7f26e0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26fa395640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26e0003b40 0x7f26e0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26fa395640 1 -- 172.21.15.146:0/1166360629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e0004800 msgr2=0x7f26e0004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26fa395640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e0004800 0x7f26e0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.541+0000 7f26fa395640 1 -- 172.21.15.146:0/1166360629 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e0003520 con 0x7f26e0005150 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.542+0000 7fe64959a640 1 -- 172.21.15.146:0/2288716757 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0b4070 con 0x7fe644048630 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.542+0000 7fe64959a640 1 -- 172.21.15.146:0/2288716757 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c0a7ab0 con 0x7fe644048630 2024-04-25T15:20:11.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.542+0000 7fe64959a640 1 -- 172.21.15.146:0/2288716757 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0af440 con 0x7fe644048630 2024-04-25T15:20:11.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.542+0000 7f26fa395640 1 -- 172.21.15.146:0/1166360629 reap_dead start 2024-04-25T15:20:11.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f26fa395640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26e000b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f26f4000b60 tx=0x7f26e0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c4bbf8310e71c32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f59812eb640 1 Processor -- start 2024-04-25T15:20:11.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f8d129ce640 1 Processor -- start 2024-04-25T15:20:11.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f8d129ce640 1 -- start start 2024-04-25T15:20:11.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f406a0e0 con 0x7f26e0005150 2024-04-25T15:20:11.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f40ab070 con 0x7f26e0005150 2024-04-25T15:20:11.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f26f8b92640 1 -- 172.21.15.146:0/1166360629 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f406e2d0 con 0x7f26e0005150 2024-04-25T15:20:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7fe84aff8640 1 Processor -- start 2024-04-25T15:20:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7fe84aff8640 1 -- start start 2024-04-25T15:20:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7fe64b59e640 1 -- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048630 msgr2=0x7fe6440ebdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7fe64b59e640 1 --2- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048630 0x7fe6440ebdb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe64c052040 tx=0x7fe64c0a6eb0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7f493c7c7640 1 Processor -- start 2024-04-25T15:20:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.543+0000 7f59812eb640 1 -- start start 2024-04-25T15:20:11.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fe64b59e640 1 -- 172.21.15.146:0/2288716757 shutdown_connections 2024-04-25T15:20:11.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fe64b59e640 1 --2- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644006aa0 0x7fe644006eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fe64b59e640 1 --2- 172.21.15.146:0/2288716757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048630 0x7fe6440ebdb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fe64b59e640 1 -- 172.21.15.146:0/2288716757 >> 172.21.15.146:0/2288716757 conn(0x7fe644055140 msgr2=0x7fe644055550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7f6305822640 1 Processor -- start 2024-04-25T15:20:11.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7f6305822640 1 -- start start 2024-04-25T15:20:11.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7f493c7c7640 1 -- start start 2024-04-25T15:20:11.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fd064b10640 1 Processor -- start 2024-04-25T15:20:11.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fe64b59e640 1 -- 172.21.15.146:0/2288716757 shutdown_connections 2024-04-25T15:20:11.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7f26fab96640 1 -- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 msgr2=0x7f26e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7f26fab96640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26e000b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f26f4000b60 tx=0x7f26e0005830 comp rx=0 tx=0).stop 2024-04-25T15:20:11.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fd064b10640 1 -- start start 2024-04-25T15:20:11.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.544+0000 7fe64b59e640 1 -- 172.21.15.146:0/2288716757 wait complete. 2024-04-25T15:20:11.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f59812eb640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c0ce2a0 0x7f597c15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f8d129ce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d0402fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f59812eb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f597c0d1f00 0x7f597c09e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f59812eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f597c164410 0x7f597c09ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f59812eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f597c0af340 con 0x7f597c164410 2024-04-25T15:20:11.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f59812eb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f597c164820 con 0x7f597c0d1f00 2024-04-25T15:20:11.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f59812eb640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f597c0af4c0 con 0x7f597c0ce2a0 2024-04-25T15:20:11.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7f8d129ce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d040500b0 0x7f8d040d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f8d129ce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 0x7f8d040314a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f597bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c0ce2a0 0x7f597c15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f597bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c0ce2a0 0x7f597c15d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39774/0 (socket says 172.21.15.146:39774) 2024-04-25T15:20:11.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f597bfff640 1 -- 172.21.15.146:0/1706604958 learned_addr learned my addr 172.21.15.146:0/1706604958 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f5980aea640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f597c164410 0x7f597c09ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f8d111cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d040500b0 0x7f8d040d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f597b7fe640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f597c0d1f00 0x7f597c09e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7fe84aff8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe844162d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.545+0000 7fe84aff8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe844165810 0x7fe8441fdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe84aff8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 0x7fe8440c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe849595640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 0x7fe8440c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe849595640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 0x7fe8440c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60098/0 (socket says 172.21.15.146:60098) 2024-04-25T15:20:11.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe843fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe844165810 0x7fe8441fdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f493c7c7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 0x7f49340c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f493c7c7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f8d111cb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d040500b0 0x7f8d040d21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45430/0 (socket says 172.21.15.146:45430) 2024-04-25T15:20:11.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f8d119cc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d0402fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe849595640 1 -- 172.21.15.146:0/1288882315 learned_addr learned my addr 172.21.15.146:0/1288882315 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f493a563640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 0x7f49340c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe8440cf8e0 con 0x7fe84406e6e0 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f493a563640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 0x7f49340c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39798/0 (socket says 172.21.15.146:39798) 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f493a563640 1 -- 172.21.15.146:0/1367001172 learned_addr learned my addr 172.21.15.146:0/1367001172 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f5980aea640 1 -- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c0ce2a0 msgr2=0x7f597c15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f5980aea640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c0ce2a0 0x7f597c15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f5980aea640 1 -- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f597c0d1f00 msgr2=0x7f597c09e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f5980aea640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f597c0d1f00 0x7f597c09e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f5980aea640 1 -- 172.21.15.146:0/1706604958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c15e6d0 con 0x7f597c164410 2024-04-25T15:20:11.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f8d121cd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 0x7f8d040314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49340d2490 0x7f493415f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f8d119cc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d0402fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39786/0 (socket says 172.21.15.146:39786) 2024-04-25T15:20:11.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f4939d62640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305822640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 0x7f63001607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f26fab96640 1 -- 172.21.15.146:0/1166360629 shutdown_connections 2024-04-25T15:20:11.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7f26fab96640 1 --2- 172.21.15.146:0/1166360629 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26e000b9c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f26fab96640 1 -- 172.21.15.146:0/1166360629 >> 172.21.15.146:0/1166360629 conn(0x7f26ec03ac50 msgr2=0x7f26ec03b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f8d121cd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 0x7f8d040314a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60114/0 (socket says 172.21.15.146:60114) 2024-04-25T15:20:11.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe8440cf760 con 0x7fe844165810 2024-04-25T15:20:11.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe8440b6560 con 0x7fe834004800 2024-04-25T15:20:11.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe848d94640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe844162d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe849595640 1 -- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 msgr2=0x7fe844162d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.546+0000 7fe849595640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe844162d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305822640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63000b5dd0 0x7f6300160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd064b10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd058006920 0x7fd0580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe849595640 1 -- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe844165810 msgr2=0x7fe8441fdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe849595640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe844165810 0x7fe8441fdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe849595640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe844166a00 con 0x7fe84406e6e0 2024-04-25T15:20:11.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305822640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f63000ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6304820640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 0x7f63001607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6304820640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 0x7f63001607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45448/0 (socket says 172.21.15.146:45448) 2024-04-25T15:20:11.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd064b10640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd0580e2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6304820640 1 -- 172.21.15.146:0/1656268498 learned_addr learned my addr 172.21.15.146:0/1656268498 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6300168380 con 0x7f6300167df0 2024-04-25T15:20:11.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6300168200 con 0x7f6300070530 2024-04-25T15:20:11.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f26fab96640 1 -- 172.21.15.146:0/1166360629 shutdown_connections 2024-04-25T15:20:11.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f26fab96640 1 -- 172.21.15.146:0/1166360629 wait complete. 2024-04-25T15:20:11.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe849595640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 0x7fe8440c9380 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe8440cdca0 tx=0x7fe83c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f630006daa0 con 0x7f63000b5dd0 2024-04-25T15:20:11.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd05f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd058006920 0x7fd0580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd05f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd058006920 0x7fd0580ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45458/0 (socket says 172.21.15.146:45458) 2024-04-25T15:20:11.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd05f7fe640 1 -- 172.21.15.146:0/2625398202 learned_addr learned my addr 172.21.15.146:0/2625398202 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f62fffff640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63000b5dd0 0x7f6300160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0580dfdd0 0x7fd05804c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd058053a90 con 0x7fd0580dfdd0 2024-04-25T15:20:11.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f5980aea640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f597c164410 0x7f597c09ec20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5974007d20 tx=0x7f597400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f6305021640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f63000ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f8d129ce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0404f340 con 0x7f8d040e47a0 2024-04-25T15:20:11.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f8d129ce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d040504c0 con 0x7f8d040500b0 2024-04-25T15:20:11.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f8d129ce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8d040e4bb0 con 0x7f8d04006090 2024-04-25T15:20:11.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f8d121cd640 1 -- 172.21.15.146:0/3629093895 learned_addr learned my addr 172.21.15.146:0/3629093895 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c010df0 con 0x7fe84406e6e0 2024-04-25T15:20:11.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe844163270 con 0x7fe84406e6e0 2024-04-25T15:20:11.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84406f620 con 0x7fe84406e6e0 2024-04-25T15:20:11.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7fd05effd640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd0580e2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f62fffff640 1 -- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 msgr2=0x7f63001607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f62fffff640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 0x7f63001607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f62fffff640 1 -- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 msgr2=0x7f63000ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f62fffff640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f63000ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f62fffff640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630015e900 con 0x7f63000b5dd0 2024-04-25T15:20:11.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd05803a940 con 0x7fd058006920 2024-04-25T15:20:11.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd05803ac20 con 0x7fd05804f900 2024-04-25T15:20:11.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f6304820640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 0x7f63001607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f493ad64640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49340d2490 0x7f493415f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.547+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f493406ecd0 con 0x7f49340d2490 2024-04-25T15:20:11.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f493406ea10 con 0x7f49340b72a0 2024-04-25T15:20:11.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f493406efb0 con 0x7f49340b1e30 2024-04-25T15:20:11.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f62fffff640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63000b5dd0 0x7f6300160ce0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f63000741d0 tx=0x7f62f00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fe64b59e640 1 Processor -- start 2024-04-25T15:20:11.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fe64b59e640 1 -- start start 2024-04-25T15:20:11.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f8d121cd640 1 -- 172.21.15.146:0/3629093895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 msgr2=0x7f8d0402fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe83c01c070 con 0x7fe84406e6e0 2024-04-25T15:20:11.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c021660 con 0x7fe84406e6e0 2024-04-25T15:20:11.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f8d121cd640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d0402fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f8d121cd640 1 -- 172.21.15.146:0/3629093895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d040500b0 msgr2=0x7f8d040d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f8d121cd640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d040500b0 0x7f8d040d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd05ffff640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0580dfdd0 0x7fd05804c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f8d121cd640 1 -- 172.21.15.146:0/3629093895 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d040dea20 con 0x7f8d040e47a0 2024-04-25T15:20:11.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd05effd640 1 -- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd058006920 msgr2=0x7fd0580ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd05effd640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd058006920 0x7fd0580ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd05effd640 1 -- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0580dfdd0 msgr2=0x7fd05804c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7fd05effd640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0580dfdd0 0x7fd05804c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7fd05effd640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0580e3000 con 0x7fd05804f900 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f493a563640 1 -- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 msgr2=0x7f49340cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f493a563640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.548+0000 7f493a563640 1 -- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49340d2490 msgr2=0x7f493415f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f493a563640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49340d2490 0x7f493415f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f493a563640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49341fe360 con 0x7f49340b1e30 2024-04-25T15:20:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0041030 con 0x7f63000b5dd0 2024-04-25T15:20:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f003d4e0 con 0x7f63000b5dd0 2024-04-25T15:20:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0041030 con 0x7f63000b5dd0 2024-04-25T15:20:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f8d121cd640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 0x7f8d040314a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800e8b0 tx=0x7f8d0800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630015ebe0 con 0x7f63000b5dd0 2024-04-25T15:20:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f493a563640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 0x7f49340c8840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f493000eac0 tx=0x7f493000ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f630015f1a0 con 0x7f63000b5dd0 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7fd05effd640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd0580e2540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd06006aed0 tx=0x7fd06008cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060075640 con 0x7fd05804f900 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06006d6d0 con 0x7fd05804f900 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06006da80 con 0x7fd05804f900 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08017070 con 0x7f8d040e47a0 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d08004020 con 0x7f8d040e47a0 2024-04-25T15:20:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d080043f0 con 0x7f8d040e47a0 2024-04-25T15:20:11.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930019480 con 0x7f49340b1e30 2024-04-25T15:20:11.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4930019620 con 0x7f49340b1e30 2024-04-25T15:20:11.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49300199f0 con 0x7f49340b1e30 2024-04-25T15:20:11.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.549+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49341fe640 con 0x7f49340b1e30 2024-04-25T15:20:11.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49341fec00 con 0x7f49340b1e30 2024-04-25T15:20:11.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0580486e0 con 0x7fd05804f900 2024-04-25T15:20:11.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd058048ca0 con 0x7fd05804f900 2024-04-25T15:20:11.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d04039030 con 0x7f8d040e47a0 2024-04-25T15:20:11.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d040395f0 con 0x7f8d040e47a0 2024-04-25T15:20:11.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597400edd0 con 0x7f597c164410 2024-04-25T15:20:11.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5974003990 con 0x7f597c164410 2024-04-25T15:20:11.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5974003d80 con 0x7f597c164410 2024-04-25T15:20:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597c09f460 con 0x7f597c164410 2024-04-25T15:20:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597c09fa20 con 0x7f597c164410 2024-04-25T15:20:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.550+0000 7f26fab96640 1 Processor -- start 2024-04-25T15:20:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.552+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d04005570 con 0x7f8d040e47a0 2024-04-25T15:20:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.551+0000 7f26fab96640 1 -- start start 2024-04-25T15:20:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.552+0000 7f62e6ffd640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6300050740 con 0x7f63000b5dd0 2024-04-25T15:20:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.551+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4934050740 con 0x7f49340b1e30 2024-04-25T15:20:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f26fab96640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26ec0d42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8d0800c940 con 0x7f8d040e47a0 2024-04-25T15:20:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd058004860 con 0x7fd05804f900 2024-04-25T15:20:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.552+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597c0b0ff0 con 0x7f597c164410 2024-04-25T15:20:11.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f26fab96640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26ec0d4ed0 0x7f26ec0d93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f26fab96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 0x7f26ec0c0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f26fab96640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec0076a0 con 0x7f26ec0d4ed0 2024-04-25T15:20:11.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f26fab96640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec03cf10 con 0x7f26ec0d98f0 2024-04-25T15:20:11.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7f26fab96640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec006d50 con 0x7f26e0005150 2024-04-25T15:20:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe83c021800 con 0x7fe84406e6e0 2024-04-25T15:20:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe824005290 con 0x7fe84406e6e0 2024-04-25T15:20:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.553+0000 7fe8417fa640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8280744b0 0x7fe8280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fe64b59e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 0x7fe6440d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 0x7f26ec0c0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 0x7f26ec0c0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45474/0 (socket says 172.21.15.146:45474) 2024-04-25T15:20:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 -- 172.21.15.146:0/2251158638 learned_addr learned my addr 172.21.15.146:0/2251158638 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe83c05c420 con 0x7fe84406e6e0 2024-04-25T15:20:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26f9393640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26ec0d4ed0 0x7f26ec0d93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f8d027fc640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8cf00745b0 0x7f8cf0076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd060094030 con 0x7fd05804f900 2024-04-25T15:20:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fe64a59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 0x7fe6440d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 -- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 msgr2=0x7f26ec0d42e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:20:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26ec0d42e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 -- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26ec0d4ed0 msgr2=0x7f26ec0d93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26ec0d4ed0 0x7f26ec0d93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 -- 172.21.15.146:0/2251158638 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec0d85f0 con 0x7f26ec0d98f0 2024-04-25T15:20:11.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fe64a59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 0x7fe6440d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60160/0 (socket says 172.21.15.146:60160) 2024-04-25T15:20:11.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8d08095610 con 0x7f8d040e47a0 2024-04-25T15:20:11.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26f9393640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26ec0d4ed0 0x7f26ec0d93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:11.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fe64a59c640 1 -- 172.21.15.146:0/541330679 learned_addr learned my addr 172.21.15.146:0/541330679 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:11.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f62f003d690 con 0x7f63000b5dd0 2024-04-25T15:20:11.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f62fd7fa640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f62ec0748e0 0x7f62ec076df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f62f0042070 con 0x7f63000b5dd0 2024-04-25T15:20:11.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048100 0x7fe64404e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.554+0000 7f26fa395640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 0x7f26ec0c0b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f26f406aed0 tx=0x7f26ec03cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644048630 0x7fe6440389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe644050980 con 0x7fe644006aa0 2024-04-25T15:20:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe644050800 con 0x7fe644048100 2024-04-25T15:20:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe6440073f0 con 0x7fe644048630 2024-04-25T15:20:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f597401b020 con 0x7f597c164410 2024-04-25T15:20:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4930019b90 con 0x7f49340b1e30 2024-04-25T15:20:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe649d9b640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048100 0x7fe64404e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe848d94640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8280744b0 0x7fe8280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe64ad9d640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644048630 0x7fe6440389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f5978ff9640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5968074580 0x7f5968076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5974095590 con 0x7f597c164410 2024-04-25T15:20:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f597bfff640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5968074580 0x7f5968076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe649d9b640 1 -- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644048630 msgr2=0x7fe6440389e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe649d9b640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644048630 0x7fe6440389e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe649d9b640 1 -- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 msgr2=0x7fe6440d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe649d9b640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 0x7fe6440d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7fe649d9b640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6440d2af0 con 0x7fe644048100 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f8d119cc640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8cf00745b0 0x7f8cf0076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f492affd640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4918074580 0x7f4918076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4930096010 con 0x7f49340b1e30 2024-04-25T15:20:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7f6304820640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f62ec0748e0 0x7f62ec076df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe649d9b640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048100 0x7fe64404e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe64c06bfc0 tx=0x7fe64c075f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe64a59c640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 0x7fe6440d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0b4050 con 0x7fe644048100 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c0b5040 con 0x7fe644048100 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c068c20 con 0x7fe644048100 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6440e56b0 con 0x7fe644048100 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f40be070 con 0x7f26ec0d98f0 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f4076290 con 0x7f26ec0d98f0 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec0d88d0 con 0x7f26ec0d98f0 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f406cb90 con 0x7f26ec0d98f0 2024-04-25T15:20:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.555+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec0d5310 con 0x7f26ec0d98f0 2024-04-25T15:20:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6440e5c70 con 0x7fe644048100 2024-04-25T15:20:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7f8d119cc640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8cf00745b0 0x7f8cf0076ac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc034000 tx=0x7f8cfc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fe848d94640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8280744b0 0x7fe8280769c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe830000fd0 tx=0x7fe830009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7f597bfff640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5968074580 0x7f5968076a90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f59700097a0 tx=0x7f597002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.557+0000 7f6304820640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f62ec0748e0 0x7f62ec076df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006f40 tx=0x7f62f4009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7f4939d62640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4918074580 0x7f4918076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.556+0000 7fd04bfff640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd044074580 0x7fd044076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.557+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd060111270 con 0x7fd05804f900 2024-04-25T15:20:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.557+0000 7fd05f7fe640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd044074580 0x7fd044076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.557+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec0059c0 con 0x7f26ec0d98f0 2024-04-25T15:20:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.557+0000 7f4939d62640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4918074580 0x7f4918076a90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f492c034000 tx=0x7f492c033040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7fd05f7fe640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd044074580 0x7fd044076a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd04c000f60 tx=0x7fd04c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe64c0c2020 con 0x7fe644048100 2024-04-25T15:20:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f26f40c1070 con 0x7f26ec0d98f0 2024-04-25T15:20:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7fe63affd640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe62c074500 0x7fe62c076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7f26ea7fc640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f26d8074500 0x7f26d8076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe64c131970 con 0x7fe644048100 2024-04-25T15:20:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7fe64a59c640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe62c074500 0x7fe62c076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7f26f9b94640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f26d8074500 0x7f26d8076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.558+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f26f4131910 con 0x7f26ec0d98f0 2024-04-25T15:20:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.559+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6440e5440 con 0x7fe644048100 2024-04-25T15:20:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.559+0000 7f26f9b94640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f26d8074500 0x7f26d8076a10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f26ec010030 tx=0x7f26e4000f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.559+0000 7fe64a59c640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe62c074500 0x7fe62c076a10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe640009d50 tx=0x7fe6400090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.559+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62f007f110 con 0x7f63000b5dd0 2024-04-25T15:20:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.561+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d0805f480 con 0x7f8d040e47a0 2024-04-25T15:20:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.564+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26f40b6040 con 0x7f26ec0d98f0 2024-04-25T15:20:11.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.564+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0600db0e0 con 0x7fd05804f900 2024-04-25T15:20:11.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.567+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe844163270 con 0x7fe84406e6e0 2024-04-25T15:20:11.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.569+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6440d2af0 con 0x7fe644048100 2024-04-25T15:20:11.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.569+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4930060560 con 0x7f49340b1e30 2024-04-25T15:20:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.581+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f597405f400 con 0x7f597c164410 2024-04-25T15:20:11.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.974+0000 7f62e6ffd640 1 -- 172.21.15.146:0/1656268498 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f630015e690 con 0x7f63000b5dd0 2024-04-25T15:20:11.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.974+0000 7f62fd7fa640 1 -- 172.21.15.146:0/1656268498 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f62f0083410 con 0x7f63000b5dd0 2024-04-25T15:20:11.975 INFO:teuthology.orchestra.run.smithi146.stdout:55834574856 2024-04-25T15:20:11.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.976+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8d0404c4f0 con 0x7f8d040e47a0 2024-04-25T15:20:11.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.977+0000 7f8d027fc640 1 -- 172.21.15.146:0/3629093895 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8d08063780 con 0x7f8d040e47a0 2024-04-25T15:20:11.978 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f62ec0748e0 msgr2=0x7f62ec076df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f62ec0748e0 0x7f62ec076df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006f40 tx=0x7f62f4009240 comp rx=0 tx=0).stop 2024-04-25T15:20:11.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63000b5dd0 msgr2=0x7f6300160ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63000b5dd0 0x7f6300160ce0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f63000741d0 tx=0x7f62f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 shutdown_connections 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f62ec0748e0 0x7f62ec076df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6300167df0 0x7f63000ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63000b5dd0 0x7f6300160ce0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 --2- 172.21.15.146:0/1656268498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6300070530 0x7f63001607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.978+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 >> 172.21.15.146:0/1656268498 conn(0x7f63000732b0 msgr2=0x7f630006cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 shutdown_connections 2024-04-25T15:20:11.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f6305822640 1 -- 172.21.15.146:0/1656268498 wait complete. 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8cf00745b0 msgr2=0x7f8cf0076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8cf00745b0 0x7f8cf0076ac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc034000 tx=0x7f8cfc02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 msgr2=0x7f8d040314a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 0x7f8d040314a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800e8b0 tx=0x7f8d0800ed70 comp rx=0 tx=0).stop 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 shutdown_connections 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8cf00745b0 0x7f8cf0076ac0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040e47a0 0x7f8d040314a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d040500b0 0x7f8d040d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 --2- 172.21.15.146:0/3629093895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d04006090 0x7f8d0402fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 >> 172.21.15.146:0/3629093895 conn(0x7f8d04054cf0 msgr2=0x7f8d04055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 shutdown_connections 2024-04-25T15:20:11.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.981+0000 7f8d129ce640 1 -- 172.21.15.146:0/3629093895 wait complete. 2024-04-25T15:20:11.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.986+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f26ec056dc0 con 0x7f26ec0d98f0 2024-04-25T15:20:11.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.987+0000 7f26ea7fc640 1 -- 172.21.15.146:0/2251158638 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f26f40fbe40 con 0x7f26ec0d98f0 2024-04-25T15:20:11.988 INFO:teuthology.orchestra.run.smithi146.stdout:55834574857 2024-04-25T15:20:11.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.988+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd058005570 con 0x7fd05804f900 2024-04-25T15:20:11.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.989+0000 7fd04bfff640 1 -- 172.21.15.146:0/2625398202 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd0600df3e0 con 0x7fd05804f900 2024-04-25T15:20:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.992+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe824005850 con 0x7fe84406e6e0 2024-04-25T15:20:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.992+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe6440059c0 con 0x7fe644048100 2024-04-25T15:20:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f26d8074500 msgr2=0x7f26d8076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f26d8074500 0x7f26d8076a10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f26ec010030 tx=0x7f26e4000f50 comp rx=0 tx=0).stop 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 msgr2=0x7f26ec0c0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 0x7f26ec0c0b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f26f406aed0 tx=0x7f26ec03cb20 comp rx=0 tx=0).stop 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 shutdown_connections 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f26d8074500 0x7f26d8076a10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26ec0d98f0 0x7f26ec0c0b90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26ec0d4ed0 0x7f26ec0d93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 --2- 172.21.15.146:0/2251158638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26e0005150 0x7f26ec0d42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.991+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 >> 172.21.15.146:0/2251158638 conn(0x7f26ec03ac50 msgr2=0x7f26ec0e3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.992+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 shutdown_connections 2024-04-25T15:20:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.992+0000 7f26fab96640 1 -- 172.21.15.146:0/2251158638 wait complete. 2024-04-25T15:20:11.995 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd044074580 msgr2=0x7fd044076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd044074580 0x7fd044076a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd04c000f60 tx=0x7fd04c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f49340cdf00 con 0x7f49340b1e30 2024-04-25T15:20:11.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 msgr2=0x7fd0580e2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd0580e2540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd06006aed0 tx=0x7fd06008cc10 comp rx=0 tx=0).stop 2024-04-25T15:20:11.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 shutdown_connections 2024-04-25T15:20:11.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd044074580 0x7fd044076a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0580dfdd0 0x7fd05804c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd05804f900 0x7fd0580e2540 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 --2- 172.21.15.146:0/2625398202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd058006920 0x7fd0580ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:11.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 >> 172.21.15.146:0/2625398202 conn(0x7fd058054cf0 msgr2=0x7fd058039380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:11.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 shutdown_connections 2024-04-25T15:20:11.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fe8417fa640 1 -- 172.21.15.146:0/1288882315 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe844163270 con 0x7fe84406e6e0 2024-04-25T15:20:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.993+0000 7fd064b10640 1 -- 172.21.15.146:0/2625398202 wait complete. 2024-04-25T15:20:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.994+0000 7fe63affd640 1 -- 172.21.15.146:0/541330679 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe64c0fbda0 con 0x7fe644048100 2024-04-25T15:20:11.998 INFO:teuthology.orchestra.run.smithi146.stdout:55834574857 2024-04-25T15:20:11.998 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.994+0000 7f492affd640 1 -- 172.21.15.146:0/1367001172 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f493005fe60 con 0x7f49340b1e30 2024-04-25T15:20:11.999 INFO:teuthology.orchestra.run.smithi146.stdout:55834574857 2024-04-25T15:20:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe62c074500 msgr2=0x7fe62c076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe62c074500 0x7fe62c076a10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe640009d50 tx=0x7fe6400090e0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8280744b0 msgr2=0x7fe8280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8280744b0 0x7fe8280769c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe830000fd0 tx=0x7fe830009040 comp rx=0 tx=0).stop 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 msgr2=0x7fe8440c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 0x7fe8440c9380 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe8440cdca0 tx=0x7fe83c010590 comp rx=0 tx=0).stop 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 shutdown_connections 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe8280744b0 0x7fe8280769c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe84406e6e0 0x7fe8440c9380 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe844165810 0x7fe8441fdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 --2- 172.21.15.146:0/1288882315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe834004800 0x7fe844162d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 >> 172.21.15.146:0/1288882315 conn(0x7fe8440b51b0 msgr2=0x7fe8440cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:12.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4918074580 msgr2=0x7f4918076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4918074580 0x7f4918076a90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f492c034000 tx=0x7f492c033040 comp rx=0 tx=0).stop 2024-04-25T15:20:12.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048100 msgr2=0x7fe64404e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.998+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048100 0x7fe64404e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe64c06bfc0 tx=0x7fe64c075f60 comp rx=0 tx=0).stop 2024-04-25T15:20:12.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 msgr2=0x7f49340c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 0x7f49340c8840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f493000eac0 tx=0x7f493000ef80 comp rx=0 tx=0).stop 2024-04-25T15:20:12.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 shutdown_connections 2024-04-25T15:20:12.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4918074580 0x7f4918076a90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 shutdown_connections 2024-04-25T15:20:12.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe84aff8640 1 -- 172.21.15.146:0/1288882315 wait complete. 2024-04-25T15:20:12.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49340d2490 0x7f493415f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 shutdown_connections 2024-04-25T15:20:12.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49340b72a0 0x7f49340cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 --2- 172.21.15.146:0/1367001172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49340b1e30 0x7f49340c8840 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 >> 172.21.15.146:0/1367001172 conn(0x7f493409b2b0 msgr2=0x7f49340cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:12.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe62c074500 0x7fe62c076a10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe644048630 0x7fe6440389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 shutdown_connections 2024-04-25T15:20:12.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7f493c7c7640 1 -- 172.21.15.146:0/1367001172 wait complete. 2024-04-25T15:20:12.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe644048100 0x7fe64404e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 --2- 172.21.15.146:0/541330679 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe644006aa0 0x7fe6440d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 >> 172.21.15.146:0/541330679 conn(0x7fe644055140 msgr2=0x7fe644005de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:12.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:11.999+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 shutdown_connections 2024-04-25T15:20:12.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.000+0000 7fe64b59e640 1 -- 172.21.15.146:0/541330679 wait complete. 2024-04-25T15:20:12.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2024-04-25T15:20:12.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-04-25T15:20:12.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.005+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f597c09f160 con 0x7f597c164410 2024-04-25T15:20:12.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.006+0000 7f5978ff9640 1 -- 172.21.15.146:0/1706604958 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5974063700 con 0x7f597c164410 2024-04-25T15:20:12.009 INFO:teuthology.orchestra.run.smithi146.stdout:55834574857 2024-04-25T15:20:12.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5968074580 msgr2=0x7f5968076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5968074580 0x7f5968076a90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f59700097a0 tx=0x7f597002e000 comp rx=0 tx=0).stop 2024-04-25T15:20:12.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f597c164410 msgr2=0x7f597c09ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f597c164410 0x7f597c09ec20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5974007d20 tx=0x7f597400e4b0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 shutdown_connections 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5968074580 0x7f5968076a90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f597c164410 0x7f597c09ec20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f597c0d1f00 0x7f597c09e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 --2- 172.21.15.146:0/1706604958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f597c0ce2a0 0x7f597c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.010+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 >> 172.21.15.146:0/1706604958 conn(0x7f597c09b2b0 msgr2=0x7f597c1630f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.011+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 shutdown_connections 2024-04-25T15:20:12.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:12.011+0000 7f59812eb640 1 -- 172.21.15.146:0/1706604958 wait complete. 2024-04-25T15:20:12.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-04-25T15:20:12.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.5 2024-04-25T15:20:12.019 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.4 2024-04-25T15:20:12.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.3 2024-04-25T15:20:12.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2024-04-25T15:20:12.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2024-04-25T15:20:13.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:20:13.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:20:13.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:20:13.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:20:13.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:20:13.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:20:13.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:20:13.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:20:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.190+0000 7f776e596640 1 Processor -- start 2024-04-25T15:20:13.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.190+0000 7f776e596640 1 -- start start 2024-04-25T15:20:13.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.194+0000 7f776e596640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f77680b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.194+0000 7f776e596640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 0x7f77680c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7f776cb33640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 0x7f77680c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7fa0727be640 1 Processor -- start 2024-04-25T15:20:13.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7f776cb33640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 0x7f77680c4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60172/0 (socket says 172.21.15.146:60172) 2024-04-25T15:20:13.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7f776cb33640 1 -- 172.21.15.146:0/2002899373 learned_addr learned my addr 172.21.15.146:0/2002899373 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77680c5410 con 0x7f776806c960 2024-04-25T15:20:13.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f776806d0e0 con 0x7f77680c4ac0 2024-04-25T15:20:13.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f776806d3c0 con 0x7f77681654d0 2024-04-25T15:20:13.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.198+0000 7f77677fe640 1 --2- 172.21.15.146:0/2002899373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f77680b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.199+0000 7f77677fe640 1 -- 172.21.15.146:0/2002899373 >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 msgr2=0x7f77680c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.199+0000 7f77677fe640 1 --2- 172.21.15.146:0/2002899373 >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 0x7f77680c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.199+0000 7f77677fe640 1 -- 172.21.15.146:0/2002899373 >> v1:172.21.15.146:6789/0 conn(0x7f776806c960 legacy=0x7f77680b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.200+0000 7f77677fe640 1 -- 172.21.15.146:0/2002899373 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77680d1a90 con 0x7f77681654d0 2024-04-25T15:20:13.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.200+0000 7f776cb33640 1 --2- 172.21.15.146:0/2002899373 >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 0x7f77680c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:13.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.197+0000 7fa0727be640 1 -- start start 2024-04-25T15:20:13.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.201+0000 7fa0727be640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa06c159960 0x7fa06c0705d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.201+0000 7fa0727be640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c06c550 con 0x7fa06c0b0220 2024-04-25T15:20:13.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.203+0000 7f77677fe640 1 --2- 172.21.15.146:0/2002899373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f77680b1000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f77580097a0 tx=0x7f775802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a191af74c0e80818 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.201+0000 7fa0727be640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0b0a00 con 0x7fa06c15f920 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.201+0000 7fa0727be640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa06c06c810 con 0x7fa06c159960 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.201+0000 7fa06b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa06c0b0220 0x7fa06c0b4430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40624/0 (socket says 172.21.15.146:40624) 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.202+0000 7fa06b7fe640 1 -- 172.21.15.146:0/2536718216 learned_addr learned my addr 172.21.15.146:0/2536718216 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.202+0000 7fa06bfff640 1 --2- 172.21.15.146:0/2536718216 >> v2:172.21.15.177:3300/0 conn(0x7fa06c159960 0x7fa06c0705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.203+0000 7fa06bfff640 1 -- 172.21.15.146:0/2536718216 >> v1:172.21.15.177:6789/0 conn(0x7fa06c15f920 legacy=0x7fa06c15fd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.203+0000 7fa06bfff640 1 -- 172.21.15.146:0/2536718216 >> v1:172.21.15.146:6789/0 conn(0x7fa06c0b0220 legacy=0x7fa06c0b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.203+0000 7fa06bfff640 1 -- 172.21.15.146:0/2536718216 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c06caf0 con 0x7fa06c159960 2024-04-25T15:20:13.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.204+0000 7f7766ffd640 1 -- 172.21.15.146:0/2002899373 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758037070 con 0x7f77681654d0 2024-04-25T15:20:13.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.205+0000 7f7766ffd640 1 -- 172.21.15.146:0/2002899373 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775802adb0 con 0x7f77681654d0 2024-04-25T15:20:13.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.205+0000 7fa06bfff640 1 --2- 172.21.15.146:0/2536718216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c159960 0x7fa06c0705d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa060009fb0 tx=0x7fa06002e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49564c2adbed5741 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.206+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 msgr2=0x7f77680b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.206+0000 7f776e596640 1 --2- 172.21.15.146:0/2002899373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f77680b1000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f77580097a0 tx=0x7f775802a0a0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.205+0000 7fa06affd640 1 -- 172.21.15.146:0/2536718216 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06003a070 con 0x7fa06c159960 2024-04-25T15:20:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 shutdown_connections 2024-04-25T15:20:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7f776e596640 1 --2- 172.21.15.146:0/2002899373 >> v2:172.21.15.146:3300/0 conn(0x7f77680c4ac0 0x7f77680c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7f776e596640 1 --2- 172.21.15.146:0/2002899373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f77680b1000 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 >> 172.21.15.146:0/2002899373 conn(0x7f776809b2b0 msgr2=0x7f77680b6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.205+0000 7fa06affd640 1 -- 172.21.15.146:0/2536718216 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa060033710 con 0x7fa06c159960 2024-04-25T15:20:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.205+0000 7fa06affd640 1 -- 172.21.15.146:0/2536718216 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060033ae0 con 0x7fa06c159960 2024-04-25T15:20:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 shutdown_connections 2024-04-25T15:20:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7f776e596640 1 -- 172.21.15.146:0/2002899373 wait complete. 2024-04-25T15:20:13.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7fa0727be640 1 -- 172.21.15.146:0/2536718216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c159960 msgr2=0x7fa06c0705d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.207+0000 7fa0727be640 1 --2- 172.21.15.146:0/2536718216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c159960 0x7fa06c0705d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa060009fb0 tx=0x7fa06002e5f0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.208+0000 7fa0727be640 1 -- 172.21.15.146:0/2536718216 shutdown_connections 2024-04-25T15:20:13.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.208+0000 7fa0727be640 1 --2- 172.21.15.146:0/2536718216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c159960 0x7fa06c0705d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.208+0000 7fa0727be640 1 -- 172.21.15.146:0/2536718216 >> 172.21.15.146:0/2536718216 conn(0x7fa06c0d1480 msgr2=0x7fa06c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.208+0000 7fa0727be640 1 -- 172.21.15.146:0/2536718216 shutdown_connections 2024-04-25T15:20:13.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.210+0000 7f776e596640 1 Processor -- start 2024-04-25T15:20:13.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.211+0000 7f776e596640 1 -- start start 2024-04-25T15:20:13.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.208+0000 7fa0727be640 1 -- 172.21.15.146:0/2536718216 wait complete. 2024-04-25T15:20:13.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.211+0000 7fa0727be640 1 Processor -- start 2024-04-25T15:20:13.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.213+0000 7f776e596640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f776806c960 0x7f776815fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.213+0000 7f776e596640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 0x7f7768161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.214+0000 7f776e596640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f776809ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.214+0000 7f776e596640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f77680b7270 con 0x7f776806c960 2024-04-25T15:20:13.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.214+0000 7f776e596640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77680b2740 con 0x7f77680c4ac0 2024-04-25T15:20:13.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.214+0000 7f776e596640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f77680d1fb0 con 0x7f77681654d0 2024-04-25T15:20:13.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.212+0000 7fa0727be640 1 -- start start 2024-04-25T15:20:13.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.214+0000 7fa0727be640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa06c0b0220 0x7fa06c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.214+0000 7f776cb33640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f776809ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f7767fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f776806c960 0x7f776815fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 0x7f7768161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 0x7f7768161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45516/0 (socket says 172.21.15.146:45516) 2024-04-25T15:20:13.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 -- 172.21.15.146:0/3850607986 learned_addr learned my addr 172.21.15.146:0/3850607986 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa0727be640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa06c159960 0x7fa06c1565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 -- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 msgr2=0x7f776809ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f776809ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 -- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f776806c960 msgr2=0x7f776815fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f776806c960 0x7f776815fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 -- 172.21.15.146:0/3850607986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776809ba30 con 0x7f77680c4ac0 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7f77677fe640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 0x7f7768161ac0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f77580098c0 tx=0x7f77580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7f7767fff640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f776806c960 0x7f776815fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758040020 con 0x7f77680c4ac0 2024-04-25T15:20:13.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7758038040 con 0x7f77680c4ac0 2024-04-25T15:20:13.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758037070 con 0x7f77680c4ac0 2024-04-25T15:20:13.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7f776e596640 1 -- 172.21.15.146:0/3850607986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77680cbf90 con 0x7f77680c4ac0 2024-04-25T15:20:13.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa0727be640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 0x7fa06c1f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7f776e596640 1 -- 172.21.15.146:0/3850607986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77680cc510 con 0x7f77680c4ac0 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa0727be640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c0b07b0 con 0x7fa06c0b0220 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa0727be640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c0b0630 con 0x7fa06c15f920 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa0727be640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa06c0741c0 con 0x7fa06c159960 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa070d5b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 0x7fa06c1f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa070d5b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 0x7fa06c1f55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45524/0 (socket says 172.21.15.146:45524) 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa070d5b640 1 -- 172.21.15.146:0/3402924197 learned_addr learned my addr 172.21.15.146:0/3402924197 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.215+0000 7fa06bfff640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa06c0b0220 0x7fa06c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7fa06b7fe640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa06c159960 0x7fa06c1565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7fa070d5b640 1 -- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa06c159960 msgr2=0x7fa06c1565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7fa070d5b640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa06c159960 0x7fa06c1565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7fa070d5b640 1 -- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa06c0b0220 msgr2=0x7fa06c159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7fa070d5b640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa06c0b0220 0x7fa06c159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.216+0000 7fa070d5b640 1 -- 172.21.15.146:0/3402924197 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c155870 con 0x7fa06c15f920 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.217+0000 7fa070d5b640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 0x7fa06c1f55e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa05800b440 tx=0x7fa05800b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.217+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058015020 con 0x7fa06c15f920 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.217+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa058011850 con 0x7fa06c15f920 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.217+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058011c40 con 0x7fa06c15f920 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.218+0000 7fa0727be640 1 -- 172.21.15.146:0/3402924197 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa06c15cf50 con 0x7fa06c15f920 2024-04-25T15:20:13.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.219+0000 7fa0727be640 1 -- 172.21.15.146:0/3402924197 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa06c155df0 con 0x7fa06c15f920 2024-04-25T15:20:13.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.219+0000 7f776e596640 1 -- 172.21.15.146:0/3850607986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7748005290 con 0x7f77680c4ac0 2024-04-25T15:20:13.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.221+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7758043020 con 0x7f77680c4ac0 2024-04-25T15:20:13.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.221+0000 7f7764ff9640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7754074580 0x7f7754076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.221+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f77580b5e90 con 0x7f77680c4ac0 2024-04-25T15:20:13.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.222+0000 7f7767fff640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7754074580 0x7f7754076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.222+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa058018020 con 0x7fa06c15f920 2024-04-25T15:20:13.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.222+0000 7fa0727be640 1 -- 172.21.15.146:0/3402924197 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa06c0af030 con 0x7fa06c15f920 2024-04-25T15:20:13.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.222+0000 7fa068ff9640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0500745b0 0x7fa050076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.222+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa058094e40 con 0x7fa06c15f920 2024-04-25T15:20:13.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.223+0000 7fa06bfff640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0500745b0 0x7fa050076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.224+0000 7fa06bfff640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0500745b0 0x7fa050076ac0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa060037000 tx=0x7fa060008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.224+0000 7f7767fff640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7754074580 0x7f7754076a90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f775c006fd0 tx=0x7f775c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.230+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7758080350 con 0x7f77680c4ac0 2024-04-25T15:20:13.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.230+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa06c0af030 con 0x7fa06c15f920 2024-04-25T15:20:13.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.260+0000 7f4de50cc640 1 Processor -- start 2024-04-25T15:20:13.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.260+0000 7f4de50cc640 1 -- start start 2024-04-25T15:20:13.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.262+0000 7f4de50cc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4de006c550 0x7f4de0149880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.262+0000 7f4de50cc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4de006c960 con 0x7f4de0147210 2024-04-25T15:20:13.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.262+0000 7f4de50cc640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4de014a1d0 con 0x7f4de0149dc0 2024-04-25T15:20:13.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.262+0000 7f4de50cc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4de00cc520 con 0x7f4de006c550 2024-04-25T15:20:13.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.263+0000 7f4ddf7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4de006c550 0x7f4de0149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.263+0000 7f4ddf7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4de006c550 0x7f4de0149880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60198/0 (socket says 172.21.15.146:60198) 2024-04-25T15:20:13.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.263+0000 7f4ddf7fe640 1 -- 172.21.15.146:0/1656754593 learned_addr learned my addr 172.21.15.146:0/1656754593 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.264+0000 7f4ddf7fe640 1 -- 172.21.15.146:0/1656754593 >> v1:172.21.15.177:6789/0 conn(0x7f4de0149dc0 legacy=0x7f4de00ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.264+0000 7f4ddf7fe640 1 -- 172.21.15.146:0/1656754593 >> v1:172.21.15.146:6789/0 conn(0x7f4de0147210 legacy=0x7f4de006fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.264+0000 7f4ddf7fe640 1 -- 172.21.15.146:0/1656754593 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de00704d0 con 0x7f4de006c550 2024-04-25T15:20:13.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.265+0000 7f4ddf7fe640 1 --2- 172.21.15.146:0/1656754593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0149880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009870 tx=0x7f4dd402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb890963b56e64ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.265+0000 7f353cfce640 1 Processor -- start 2024-04-25T15:20:13.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.265+0000 7f4ddeffd640 1 -- 172.21.15.146:0/1656754593 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4037070 con 0x7f4de006c550 2024-04-25T15:20:13.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.266+0000 7f4ddeffd640 1 -- 172.21.15.146:0/1656754593 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd402ada0 con 0x7f4de006c550 2024-04-25T15:20:13.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.264+0000 7feb4bfce640 1 Processor -- start 2024-04-25T15:20:13.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.264+0000 7feb4bfce640 1 -- start start 2024-04-25T15:20:13.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.266+0000 7feb4bfce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb440cd330 0x7feb440cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7f4de50cc640 1 -- 172.21.15.146:0/1656754593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 msgr2=0x7f4de0149880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7f4de50cc640 1 --2- 172.21.15.146:0/1656754593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0149880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009870 tx=0x7f4dd402a110 comp rx=0 tx=0).stop 2024-04-25T15:20:13.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.266+0000 7feb4a56b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7feb4408bcd0 0x7feb4408c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40630/0 (socket says 172.21.15.146:40630) 2024-04-25T15:20:13.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.266+0000 7feb49d6a640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7feb44163a40 0x7feb440cf210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38982/0 (socket says 172.21.15.146:38982) 2024-04-25T15:20:13.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb4a56b640 1 -- 172.21.15.146:0/79850568 learned_addr learned my addr 172.21.15.146:0/79850568 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:13.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb44070ea0 con 0x7feb4408bcd0 2024-04-25T15:20:13.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.265+0000 7f353cfce640 1 -- start start 2024-04-25T15:20:13.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7f0ec9546640 1 Processor -- start 2024-04-25T15:20:13.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f0ec9546640 1 -- start start 2024-04-25T15:20:13.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353cfce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f35380702b0 0x7f35380d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353cfce640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f353806cc60 0x7f35380cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f3535d9b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f35381645c0 0x7f353806c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:38996/0 (socket says 172.21.15.146:38996) 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f35380702b0 0x7f35380d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f3536d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f353806cc60 0x7f35380cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f35380702b0 0x7f35380d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60204/0 (socket says 172.21.15.146:60204) 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f3536d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f353806cc60 0x7f35380cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45526/0 (socket says 172.21.15.146:45526) 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f4de50cc640 1 -- 172.21.15.146:0/1656754593 shutdown_connections 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f4de50cc640 1 --2- 172.21.15.146:0/1656754593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0149880 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f4de50cc640 1 -- 172.21.15.146:0/1656754593 >> 172.21.15.146:0/1656754593 conn(0x7f4de00cde40 msgr2=0x7f4de00ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb440ccb30 con 0x7feb44163a40 2024-04-25T15:20:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7feb440cdc80 con 0x7feb440cd330 2024-04-25T15:20:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451785197 0 0) 0x7feb44070ea0 con 0x7feb4408bcd0 2024-04-25T15:20:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb30007930 con 0x7feb4408bcd0 2024-04-25T15:20:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1313455836 0 0) 0x7feb440ccb30 con 0x7feb44163a40 2024-04-25T15:20:13.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb44070ea0 con 0x7feb44163a40 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3992896355 0 0) 0x7feb30007930 con 0x7feb4408bcd0 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb440cd330 msgr2=0x7feb440cd740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb440cd330 0x7feb440cd740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.267+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 >> v1:172.21.15.177:6789/0 conn(0x7feb44163a40 legacy=0x7feb440cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb44071180 con 0x7feb4408bcd0 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7feb34001670 con 0x7feb4408bcd0 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 >> v1:172.21.15.146:6789/0 conn(0x7feb4408bcd0 legacy=0x7feb4408c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb30007f80 0x7feb300083d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb30008c10 0x7feb30009020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb30009560 0x7feb3000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feb440cdc80 con 0x7feb30009560 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb3000c2f0 con 0x7feb30008c10 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7feb3000c4a0 con 0x7feb30007f80 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb4a56b640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb30009560 0x7feb3000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb49d6a640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb30007f80 0x7feb300083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb4a56b640 1 -- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb30007f80 msgr2=0x7feb300083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb4a56b640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb30007f80 0x7feb300083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb4a56b640 1 -- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb30008c10 msgr2=0x7feb30009020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb4a56b640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb30008c10 0x7feb30009020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7feb4a56b640 1 -- 172.21.15.146:0/79850568 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb30007930 con 0x7feb30009560 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7feb49d6a640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb30007f80 0x7feb300083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7fb6805f3640 1 Processor -- start 2024-04-25T15:20:13.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7fb6805f3640 1 -- start start 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353cfce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35380cdc30 con 0x7f35381645c0 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353cfce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f35380cdf10 con 0x7f35380702b0 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.268+0000 7f353cfce640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f35380cfd50 con 0x7f353806cc60 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7f3535d9b640 1 -- 172.21.15.146:0/3339618356 learned_addr learned my addr 172.21.15.146:0/3339618356 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7f353659c640 1 -- 172.21.15.146:0/3339618356 >> v2:172.21.15.177:3300/0 conn(0x7f353806cc60 msgr2=0x7f35380cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7f353659c640 1 --2- 172.21.15.146:0/3339618356 >> v2:172.21.15.177:3300/0 conn(0x7f353806cc60 0x7f35380cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7f353659c640 1 -- 172.21.15.146:0/3339618356 >> v1:172.21.15.177:6789/0 conn(0x7f35381645c0 legacy=0x7f353806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.269+0000 7f353659c640 1 -- 172.21.15.146:0/3339618356 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35380d0030 con 0x7f35380702b0 2024-04-25T15:20:13.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec9546640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 0x7f0ec4160a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7feb4a56b640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb30009560 0x7feb3000bdb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feb440732d0 tx=0x7feb30009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e020d75597bb6f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec9546640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0ec4160f60 0x7f0ec4161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec9546640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec40cc7e0 con 0x7f0ec406c550 2024-04-25T15:20:13.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec9546640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec41618b0 con 0x7f0ec4160f60 2024-04-25T15:20:13.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec9546640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec406e0a0 con 0x7f0ec40d0420 2024-04-25T15:20:13.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f3535d9b640 1 --1- 172.21.15.146:0/3339618356 >> v1:172.21.15.177:6789/0 conn(0x7f35381645c0 0x7f353806c550 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T15:20:13.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34007ce0 con 0x7feb30009560 2024-04-25T15:20:13.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 0x7f0ec4160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb34050070 con 0x7feb30009560 2024-04-25T15:20:13.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7feb48d68640 1 -- 172.21.15.146:0/79850568 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34005d70 con 0x7feb30009560 2024-04-25T15:20:13.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec2ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 0x7f0ec4160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39854/0 (socket says 172.21.15.146:39854) 2024-04-25T15:20:13.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec2ffd640 1 -- 172.21.15.146:0/736658298 learned_addr learned my addr 172.21.15.146:0/736658298 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.270+0000 7f0ec27fc640 1 --2- 172.21.15.146:0/736658298 >> v2:172.21.15.177:3300/0 conn(0x7f0ec4160f60 0x7f0ec4161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f353659c640 1 --2- 172.21.15.146:0/3339618356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35380702b0 0x7f35380d0830 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f35380b0d10 tx=0x7f352802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6969f64c31abca57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f4de50cc640 1 -- 172.21.15.146:0/1656754593 shutdown_connections 2024-04-25T15:20:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f0ec1ffb640 1 -- 172.21.15.146:0/736658298 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247678859 0 0) 0x7f0ec40cc7e0 con 0x7f0ec406c550 2024-04-25T15:20:13.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f0ec1ffb640 1 -- 172.21.15.146:0/736658298 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eac003520 con 0x7f0ec406c550 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f353559a640 1 -- 172.21.15.146:0/3339618356 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528037070 con 0x7f35380702b0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f353559a640 1 -- 172.21.15.146:0/3339618356 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352802ad50 con 0x7f35380702b0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f353559a640 1 -- 172.21.15.146:0/3339618356 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528032660 con 0x7f35380702b0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7fb6805f3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 0x7fb678070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7fb6805f3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb67806c8c0 0x7fb67806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7fb6805f3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6780cd490 con 0x7fb6781618f0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7fb6805f3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb67815bcd0 con 0x7fb67806c8c0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7fb6805f3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb6780711d0 con 0x7fb6780cc280 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67eb90640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb67806c8c0 0x7fb67806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 0x7fb678070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 0x7fb678070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39866/0 (socket says 172.21.15.146:39866) 2024-04-25T15:20:13.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 -- 172.21.15.146:0/2024003178 learned_addr learned my addr 172.21.15.146:0/2024003178 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67db8e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb6781618f0 0x7fb678161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40646/0 (socket says 172.21.15.146:40646) 2024-04-25T15:20:13.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb30009560 msgr2=0x7feb3000bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7feb4bfce640 1 --2- 172.21.15.146:0/79850568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb30009560 0x7feb3000bdb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feb440732d0 tx=0x7feb30009c40 comp rx=0 tx=0).stop 2024-04-25T15:20:13.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f4de50cc640 1 -- 172.21.15.146:0/1656754593 wait complete. 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f0ec2ffd640 1 -- 172.21.15.146:0/736658298 >> v2:172.21.15.177:3300/0 conn(0x7f0ec4160f60 msgr2=0x7f0ec4161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f0ec2ffd640 1 --2- 172.21.15.146:0/736658298 >> v2:172.21.15.177:3300/0 conn(0x7f0ec4160f60 0x7f0ec4161370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f0ec2ffd640 1 -- 172.21.15.146:0/736658298 >> v1:172.21.15.177:6789/0 conn(0x7f0ec406c550 legacy=0x7f0ec406c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.271+0000 7f0ec2ffd640 1 -- 172.21.15.146:0/736658298 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec406e380 con 0x7f0ec40d0420 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7feb4a56b640 1 -- 172.21.15.146:0/79850568 reap_dead start 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f353cfce640 1 -- 172.21.15.146:0/3339618356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35380702b0 msgr2=0x7f35380d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f353cfce640 1 --2- 172.21.15.146:0/3339618356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35380702b0 0x7f35380d0830 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f35380b0d10 tx=0x7f352802a040 comp rx=0 tx=0).stop 2024-04-25T15:20:13.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 -- 172.21.15.146:0/2024003178 >> v2:172.21.15.146:3300/0 conn(0x7fb67806c8c0 msgr2=0x7fb67806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f353cfce640 1 -- 172.21.15.146:0/3339618356 shutdown_connections 2024-04-25T15:20:13.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f353cfce640 1 --2- 172.21.15.146:0/3339618356 >> v2:172.21.15.177:3300/0 conn(0x7f353806cc60 0x7f35380cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f353cfce640 1 --2- 172.21.15.146:0/3339618356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35380702b0 0x7f35380d0830 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f353cfce640 1 -- 172.21.15.146:0/3339618356 >> 172.21.15.146:0/3339618356 conn(0x7f35380ce0c0 msgr2=0x7f35380ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 --2- 172.21.15.146:0/2024003178 >> v2:172.21.15.146:3300/0 conn(0x7fb67806c8c0 0x7fb67806ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 -- 172.21.15.146:0/2024003178 >> v1:172.21.15.146:6789/0 conn(0x7fb6781618f0 legacy=0x7fb678161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7f0ec2ffd640 1 --2- 172.21.15.146:0/736658298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 0x7f0ec4160a20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009e50 tx=0x7f0ebc02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114eb81948f69747 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7f0ec1ffb640 1 -- 172.21.15.146:0/736658298 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc036070 con 0x7f0ec40d0420 2024-04-25T15:20:13.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7f0ec1ffb640 1 -- 172.21.15.146:0/736658298 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ebc031600 con 0x7f0ec40d0420 2024-04-25T15:20:13.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7faee65e3640 1 Processor -- start 2024-04-25T15:20:13.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.272+0000 7fb67e38f640 1 -- 172.21.15.146:0/2024003178 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6780cc880 con 0x7fb6780cc280 2024-04-25T15:20:13.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7faee65e3640 1 -- start start 2024-04-25T15:20:13.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7fb67eb90640 1 --2- 172.21.15.146:0/2024003178 >> v2:172.21.15.146:3300/0 conn(0x7fb67806c8c0 0x7fb67806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:13.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 shutdown_connections 2024-04-25T15:20:13.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 >> 172.21.15.146:0/79850568 conn(0x7feb440ce0c0 msgr2=0x7feb440ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7f353cfce640 1 -- 172.21.15.146:0/3339618356 shutdown_connections 2024-04-25T15:20:13.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 shutdown_connections 2024-04-25T15:20:13.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7feb4bfce640 1 -- 172.21.15.146:0/79850568 wait complete. 2024-04-25T15:20:13.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7f353cfce640 1 -- 172.21.15.146:0/3339618356 wait complete. 2024-04-25T15:20:13.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7fb67e38f640 1 --2- 172.21.15.146:0/2024003178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 0x7fb678070c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb670009e50 tx=0x7fb67002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c8e56399769e72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.273+0000 7f0ec1ffb640 1 -- 172.21.15.146:0/736658298 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0319d0 con 0x7f0ec40d0420 2024-04-25T15:20:13.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f0ec9546640 1 -- 172.21.15.146:0/736658298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 msgr2=0x7f0ec4160a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7fb67d38d640 1 -- 172.21.15.146:0/2024003178 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670036070 con 0x7fb6780cc280 2024-04-25T15:20:13.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7fb67d38d640 1 -- 172.21.15.146:0/2024003178 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb670031600 con 0x7fb6780cc280 2024-04-25T15:20:13.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f0ec9546640 1 --2- 172.21.15.146:0/736658298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 0x7f0ec4160a20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009e50 tx=0x7f0ebc02a680 comp rx=0 tx=0).stop 2024-04-25T15:20:13.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f0ec9546640 1 -- 172.21.15.146:0/736658298 shutdown_connections 2024-04-25T15:20:13.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7fb67d38d640 1 -- 172.21.15.146:0/2024003178 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670036070 con 0x7fb6780cc280 2024-04-25T15:20:13.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f0ec9546640 1 --2- 172.21.15.146:0/736658298 >> v2:172.21.15.177:3300/0 conn(0x7f0ec4160f60 0x7f0ec4161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f0ec9546640 1 --2- 172.21.15.146:0/736658298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec40d0420 0x7f0ec4160a20 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009e50 tx=0x7f0ebc02a680 comp rx=0 tx=0).stop 2024-04-25T15:20:13.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f0ec9546640 1 -- 172.21.15.146:0/736658298 >> 172.21.15.146:0/736658298 conn(0x7f0ec409b2b0 msgr2=0x7f0ec40b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f4de50cc640 1 Processor -- start 2024-04-25T15:20:13.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.274+0000 7f4de50cc640 1 -- start start 2024-04-25T15:20:13.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7faee65e3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7faee65e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee006f2c0 0x7faee00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7fb6805f3640 1 -- 172.21.15.146:0/2024003178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 msgr2=0x7fb678070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7fb6805f3640 1 --2- 172.21.15.146:0/2024003178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 0x7fb678070c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb670009e50 tx=0x7fb67002a5f0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7faee65e3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee009c2a0 con 0x7faee006f6d0 2024-04-25T15:20:13.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7faee65e3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7faee00d1d60 con 0x7faee00d21f0 2024-04-25T15:20:13.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7faee65e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faee009bce0 con 0x7faee006f2c0 2024-04-25T15:20:13.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7f0ec9546640 1 -- 172.21.15.146:0/736658298 shutdown_connections 2024-04-25T15:20:13.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.275+0000 7f0ec9546640 1 -- 172.21.15.146:0/736658298 wait complete. 2024-04-25T15:20:13.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60230/0 (socket says 172.21.15.146:60230) 2024-04-25T15:20:13.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 -- 172.21.15.146:0/1773916899 learned_addr learned my addr 172.21.15.146:0/1773916899 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7f353cfce640 1 Processor -- start 2024-04-25T15:20:13.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedf7fe640 1 --2- 172.21.15.146:0/1773916899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee006f2c0 0x7faee00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 -- 172.21.15.146:0/1773916899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee006f2c0 msgr2=0x7faee00b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 --2- 172.21.15.146:0/1773916899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee006f2c0 0x7faee00b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 -- 172.21.15.146:0/1773916899 >> v1:172.21.15.146:6789/0 conn(0x7faee006f6d0 legacy=0x7faee00b21b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:20:13.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7faedffff640 1 -- 172.21.15.146:0/1773916899 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee009bfc0 con 0x7faee00d21f0 2024-04-25T15:20:13.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7f353cfce640 1 -- start start 2024-04-25T15:20:13.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7f4de50cc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7f4de50cc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de0147210 0x7f4de01fa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7f4de50cc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de0149dc0 0x7f4de01ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.276+0000 7f4de50cc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4de00cc7e0 con 0x7f4de006c550 2024-04-25T15:20:13.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de0147210 0x7f4de01fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4de50cc640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4de0071040 con 0x7f4de0147210 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4de50cc640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4de0070a80 con 0x7f4de0149dc0 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de0147210 0x7f4de01fa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45558/0 (socket says 172.21.15.146:45558) 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddf7fe640 1 -- 172.21.15.146:0/3238908307 learned_addr learned my addr 172.21.15.146:0/3238908307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 -- 172.21.15.146:0/3238908307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de0149dc0 msgr2=0x7f4de01ef030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de0149dc0 0x7f4de01ef030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 -- 172.21.15.146:0/3238908307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de0147210 msgr2=0x7f4de01fa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de0147210 0x7f4de01fa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 -- 172.21.15.146:0/3238908307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de0146a70 con 0x7f4de006c550 2024-04-25T15:20:13.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7fb6805f3640 1 -- 172.21.15.146:0/2024003178 shutdown_connections 2024-04-25T15:20:13.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7fb6805f3640 1 --2- 172.21.15.146:0/2024003178 >> v2:172.21.15.146:3300/0 conn(0x7fb67806c8c0 0x7fb67806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7fb6805f3640 1 --2- 172.21.15.146:0/2024003178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6780cc280 0x7fb678070c90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7fb6805f3640 1 -- 172.21.15.146:0/2024003178 >> 172.21.15.146:0/2024003178 conn(0x7fb6780ce0c0 msgr2=0x7fb6780ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7f4ddffff640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0146200 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000c780 tx=0x7f4dd000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7fb6805f3640 1 -- 172.21.15.146:0/2024003178 shutdown_connections 2024-04-25T15:20:13.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.277+0000 7fb6805f3640 1 -- 172.21.15.146:0/2024003178 wait complete. 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0017070 con 0x7f4de006c550 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd0005270 con 0x7f4de006c550 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd000a640 con 0x7f4de006c550 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de01edf90 con 0x7f4de006c550 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7f353cfce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f353806cc60 0x7f35380ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f353cfce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35381645c0 0x7f353809bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f353cfce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 0x7f353809c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7faedffff640 1 --2- 172.21.15.146:0/1773916899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee00d21f0 0x7faee015ded0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faed0009ed0 tx=0x7faed002eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27727fbefe9f509 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f353659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f353806cc60 0x7f35380ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f353659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f353806cc60 0x7f35380ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45574/0 (socket says 172.21.15.146:45574) 2024-04-25T15:20:13.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de01ee550 con 0x7f4de006c550 2024-04-25T15:20:13.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f3535d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35381645c0 0x7f353809bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.279+0000 7f3535d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35381645c0 0x7f353809bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39900/0 (socket says 172.21.15.146:39900) 2024-04-25T15:20:13.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7feb4bfce640 1 Processor -- start 2024-04-25T15:20:13.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7feb4bfce640 1 -- start start 2024-04-25T15:20:13.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 0x7f353809c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 0x7f353809c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60258/0 (socket says 172.21.15.146:60258) 2024-04-25T15:20:13.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f353659c640 1 -- 172.21.15.146:0/3922514845 learned_addr learned my addr 172.21.15.146:0/3922514845 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f353806d280 con 0x7f353809c520 2024-04-25T15:20:13.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.278+0000 7f0ec9546640 1 Processor -- start 2024-04-25T15:20:13.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35380ccc10 con 0x7f353806cc60 2024-04-25T15:20:13.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f353809d370 con 0x7f35381645c0 2024-04-25T15:20:13.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4bfce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 0x7feb44162950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4bfce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb44167220 0x7feb4416f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4bfce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb4416f7e0 0x7feb440b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4bfce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feb44163e50 con 0x7feb4416f7e0 2024-04-25T15:20:13.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4bfce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb440ccea0 con 0x7feb4409ebe0 2024-04-25T15:20:13.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4bfce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7feb44163fd0 con 0x7feb44167220 2024-04-25T15:20:13.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7f0ec9546640 1 -- start start 2024-04-25T15:20:13.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4dd001a070 con 0x7f4de006c550 2024-04-25T15:20:13.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7faedeffd640 1 -- 172.21.15.146:0/1773916899 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed00306b0 con 0x7faee00d21f0 2024-04-25T15:20:13.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7faedeffd640 1 -- 172.21.15.146:0/1773916899 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed0030850 con 0x7faee00d21f0 2024-04-25T15:20:13.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7faedeffd640 1 -- 172.21.15.146:0/1773916899 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0030c20 con 0x7faee00d21f0 2024-04-25T15:20:13.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7fb6805f3640 1 Processor -- start 2024-04-25T15:20:13.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7fb6805f3640 1 -- start start 2024-04-25T15:20:13.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb49d6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 0x7feb44162950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb49d6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 0x7feb44162950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45590/0 (socket says 172.21.15.146:45590) 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb49d6a640 1 -- 172.21.15.146:0/822636139 learned_addr learned my addr 172.21.15.146:0/822636139 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.280+0000 7feb4a56b640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb4416f7e0 0x7feb440b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de0050740 con 0x7f4de006c550 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f3536d9d640 1 -- 172.21.15.146:0/3922514845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35381645c0 msgr2=0x7f353809bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f3536d9d640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35381645c0 0x7f353809bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f3536d9d640 1 -- 172.21.15.146:0/3922514845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f353806cc60 msgr2=0x7f35380ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f3536d9d640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f353806cc60 0x7f35380ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f3536d9d640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35380b23f0 con 0x7f353809c520 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7feb49569640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb44167220 0x7feb4416f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7feb49d6a640 1 -- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb44167220 msgr2=0x7feb4416f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7feb49d6a640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb44167220 0x7feb4416f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7feb49d6a640 1 -- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb4416f7e0 msgr2=0x7feb440b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7feb49d6a640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb4416f7e0 0x7feb440b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7feb49d6a640 1 -- 172.21.15.146:0/822636139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4416fbf0 con 0x7feb4409ebe0 2024-04-25T15:20:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 -- 172.21.15.146:0/1773916899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee00d21f0 msgr2=0x7faee015ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 --2- 172.21.15.146:0/1773916899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee00d21f0 0x7faee015ded0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faed0009ed0 tx=0x7faed002eba0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 -- 172.21.15.146:0/1773916899 shutdown_connections 2024-04-25T15:20:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 --2- 172.21.15.146:0/1773916899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee006f2c0 0x7faee00b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 --2- 172.21.15.146:0/1773916899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 -- 172.21.15.146:0/1773916899 >> 172.21.15.146:0/1773916899 conn(0x7faee00b7530 msgr2=0x7faee00b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f0ec9546640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ec406c550 0x7f0ec4150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f0ec9546640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ec4160f60 0x7f0ec4150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f0ec9546640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec4151d60 0x7f0ec40ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f0ec9546640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40d0830 con 0x7f0ec406c550 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f0ec9546640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40d0c00 con 0x7f0ec4160f60 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f0ec9546640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec40d09b0 con 0x7f0ec4151d60 2024-04-25T15:20:13.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f3536d9d640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 0x7f353809c930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f353000ee00 tx=0x7f353000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7faee65e3640 1 -- 172.21.15.146:0/1773916899 shutdown_connections 2024-04-25T15:20:13.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7faee65e3640 1 -- 172.21.15.146:0/1773916899 wait complete. 2024-04-25T15:20:13.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7feb49d6a640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 0x7feb44162950 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb3c001d10 tx=0x7feb3c0021d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.281+0000 7f4ddcff9640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4dcc074580 0x7f4dcc076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f4ddf7fe640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4dcc074580 0x7f4dcc076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4dd0094920 con 0x7f4de006c550 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.282+0000 7f4ddf7fe640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4dcc074580 0x7f4dcc076a90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4dd402a150 tx=0x7f4dd402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ec4160f60 0x7f0ec4150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec27fc640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ec4160f60 0x7f0ec4150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45592/0 (socket says 172.21.15.146:45592) 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec27fc640 1 -- 172.21.15.146:0/63265493 learned_addr learned my addr 172.21.15.146:0/63265493 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec37fe640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec4151d60 0x7f0ec40ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec2ffd640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ec406c550 0x7f0ec4150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c019070 con 0x7feb4409ebe0 2024-04-25T15:20:13.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb3c004e30 con 0x7feb4409ebe0 2024-04-25T15:20:13.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00ea00 con 0x7feb4409ebe0 2024-04-25T15:20:13.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb44166780 con 0x7feb4409ebe0 2024-04-25T15:20:13.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4409f3a0 con 0x7feb4409ebe0 2024-04-25T15:20:13.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7fb6805f3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 0x7fb678156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec37fe640 1 -- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ec4160f60 msgr2=0x7f0ec4150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec37fe640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ec4160f60 0x7f0ec4150cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec37fe640 1 -- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ec406c550 msgr2=0x7f0ec4150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec37fe640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ec406c550 0x7f0ec4150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.283+0000 7f0ec37fe640 1 -- 172.21.15.146:0/63265493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec40cb930 con 0x7f0ec4151d60 2024-04-25T15:20:13.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb6805f3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6780cc280 0x7fb6780b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb6805f3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6781618f0 0x7fb6780b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb67e38f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 0x7fb678156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb67e38f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 0x7fb678156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:39932/0 (socket says 172.21.15.146:39932) 2024-04-25T15:20:13.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb67e38f640 1 -- 172.21.15.146:0/1842512368 learned_addr learned my addr 172.21.15.146:0/1842512368 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7f0ec37fe640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec4151d60 0x7f0ec40ca880 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40cf9c0 tx=0x7f0eb4001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb67eb90640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6781618f0 0x7fb6780b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb6780cd120 con 0x7fb6781618f0 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb67db8e640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6780cc280 0x7fb6780b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6780cce40 con 0x7fb6780cc280 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb6780b0940 con 0x7fb67806c8c0 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67e38f640 1 -- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6780cc280 msgr2=0x7fb6780b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67e38f640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6780cc280 0x7fb6780b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67e38f640 1 -- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6781618f0 msgr2=0x7fb6780b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7faee65e3640 1 Processor -- start 2024-04-25T15:20:13.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67e38f640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6781618f0 0x7fb6780b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67e38f640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67809e260 con 0x7fb67806c8c0 2024-04-25T15:20:13.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4017070 con 0x7f0ec4151d60 2024-04-25T15:20:13.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec415bb90 con 0x7f0ec4151d60 2024-04-25T15:20:13.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.284+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec415be70 con 0x7f0ec4151d60 2024-04-25T15:20:13.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb40052f0 con 0x7f0ec4151d60 2024-04-25T15:20:13.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67db8e640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6780cc280 0x7fb6780b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:13.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7fb67eb90640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6781618f0 0x7fb6780b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:13.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7faee65e3640 1 -- start start 2024-04-25T15:20:13.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb2c005290 con 0x7feb4409ebe0 2024-04-25T15:20:13.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.285+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4003830 con 0x7f0ec4151d60 2024-04-25T15:20:13.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7fb67e38f640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 0x7fb678156cd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb670034040 tx=0x7fb670004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670041070 con 0x7fb67806c8c0 2024-04-25T15:20:13.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb670007990 con 0x7fb67806c8c0 2024-04-25T15:20:13.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670041070 con 0x7fb67806c8c0 2024-04-25T15:20:13.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67809be30 con 0x7fb67806c8c0 2024-04-25T15:20:13.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feb3c01c070 con 0x7feb4409ebe0 2024-04-25T15:20:13.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.286+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea4005290 con 0x7f0ec4151d60 2024-04-25T15:20:13.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.287+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6780b4650 con 0x7fb67806c8c0 2024-04-25T15:20:13.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.287+0000 7feb3a7fc640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feb280745b0 0x7feb28076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.287+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7feb3c094c80 con 0x7feb4409ebe0 2024-04-25T15:20:13.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7feb49569640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feb280745b0 0x7feb28076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0eb401a070 con 0x7f0ec4151d60 2024-04-25T15:20:13.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faee65e3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee006f2c0 0x7faee014a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faee65e3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 0x7faee014ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67806a0e0 con 0x7fb67806c8c0 2024-04-25T15:20:13.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faee65e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faedffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee006f2c0 0x7faee014a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7f0eab7fe640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0eb00744b0 0x7f0eb00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0eb40947d0 con 0x7f0ec4151d60 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faedffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee006f2c0 0x7faee014a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60280/0 (socket says 172.21.15.146:60280) 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faedf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 0x7faee014ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.288+0000 7faedf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 0x7faee014ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45606/0 (socket says 172.21.15.146:45606) 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faedffff640 1 -- 172.21.15.146:0/1084261431 learned_addr learned my addr 172.21.15.146:0/1084261431 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faee4b80640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faee65e3640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faee009b580 con 0x7faee006f2c0 2024-04-25T15:20:13.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7f0ec2ffd640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0eb00744b0 0x7f0eb00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faee65e3640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faee009b400 con 0x7faee006f6d0 2024-04-25T15:20:13.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faee65e3640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faee00b3c10 con 0x7faee00d21f0 2024-04-25T15:20:13.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faedf7fe640 1 -- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee00d21f0 msgr2=0x7faee015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faedf7fe640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faedf7fe640 1 -- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee006f2c0 msgr2=0x7faee014a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faedf7fe640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee006f2c0 0x7faee014a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7faedf7fe640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee0160c40 con 0x7faee006f6d0 2024-04-25T15:20:13.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb670044070 con 0x7fb67806c8c0 2024-04-25T15:20:13.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7fb66effd640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb65c0748b0 0x7fb65c076dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.289+0000 7feb49569640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feb280745b0 0x7feb28076ac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feb44167630 tx=0x7feb40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530018020 con 0x7f353809c520 2024-04-25T15:20:13.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4de006c960 con 0x7f4de006c550 2024-04-25T15:20:13.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353000ce20 con 0x7f353809c520 2024-04-25T15:20:13.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35300146c0 con 0x7f353809c520 2024-04-25T15:20:13.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353809b500 con 0x7f353809c520 2024-04-25T15:20:13.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7faedf7fe640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 0x7faee014ac40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faed400d520 tx=0x7faed400d9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353809bac0 con 0x7f353809c520 2024-04-25T15:20:13.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.290+0000 7fb67db8e640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb65c0748b0 0x7fb65c076dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.291+0000 7f0ec2ffd640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0eb00744b0 0x7f0eb00769c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc033000 tx=0x7f0ebc009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7fb67db8e640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb65c0748b0 0x7fb65c076dc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb678158f00 tx=0x7fb668007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed40125f0 con 0x7faee006f6d0 2024-04-25T15:20:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed4019070 con 0x7faee006f6d0 2024-04-25T15:20:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed40129b0 con 0x7faee006f6d0 2024-04-25T15:20:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7faee65e3640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee014b180 con 0x7faee006f6d0 2024-04-25T15:20:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f351c005290 con 0x7f353809c520 2024-04-25T15:20:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb3c09b080 con 0x7feb4409ebe0 2024-04-25T15:20:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.292+0000 7faee65e3640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee015baa0 con 0x7faee006f6d0 2024-04-25T15:20:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.293+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0eb405e640 con 0x7f0ec4151d60 2024-04-25T15:20:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.293+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3530005d10 con 0x7f353809c520 2024-04-25T15:20:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.293+0000 7f3526ffd640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f35180744b0 0x7f35180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.293+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3530096270 con 0x7f353809c520 2024-04-25T15:20:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.294+0000 7f353659c640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f35180744b0 0x7f35180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.294+0000 7f353659c640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f35180744b0 0x7f35180769c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3528009560 tx=0x7f352802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.295+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb670031400 con 0x7fb67806c8c0 2024-04-25T15:20:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.295+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee00d1140 con 0x7faee006f6d0 2024-04-25T15:20:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.297+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb67007f780 con 0x7fb67806c8c0 2024-04-25T15:20:13.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.297+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7faed4012b50 con 0x7faee006f6d0 2024-04-25T15:20:13.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.298+0000 7faedcff9640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faecc0745b0 0x7faecc076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.298+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faed4095d50 con 0x7faee006f6d0 2024-04-25T15:20:13.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.299+0000 7faedffff640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faecc0745b0 0x7faecc076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.299+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3530022030 con 0x7f353809c520 2024-04-25T15:20:13.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.299+0000 7faedffff640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faecc0745b0 0x7faecc076ac0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faed0000f80 tx=0x7faed0009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.301+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faee00d1140 con 0x7faee006f6d0 2024-04-25T15:20:13.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.643+0000 7f776e596640 1 -- 172.21.15.146:0/3850607986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7748005850 con 0x7f77680c4ac0 2024-04-25T15:20:13.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.643+0000 7fa0727be640 1 -- 172.21.15.146:0/3402924197 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa06c050740 con 0x7fa06c15f920 2024-04-25T15:20:13.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.644+0000 7fa068ff9640 1 -- 172.21.15.146:0/3402924197 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa05801f070 con 0x7fa06c15f920 2024-04-25T15:20:13.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.644+0000 7f7764ff9640 1 -- 172.21.15.146:0/3850607986 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f775807fc50 con 0x7f77680c4ac0 2024-04-25T15:20:13.645 INFO:teuthology.orchestra.run.smithi146.stdout:55834574859 2024-04-25T15:20:13.646 INFO:teuthology.orchestra.run.smithi146.stdout:55834574857 2024-04-25T15:20:13.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 -- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0500745b0 msgr2=0x7fa050076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0500745b0 0x7fa050076ac0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa060037000 tx=0x7fa060008030 comp rx=0 tx=0).stop 2024-04-25T15:20:13.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 -- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 msgr2=0x7fa06c1f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 0x7fa06c1f55e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa05800b440 tx=0x7fa05800b900 comp rx=0 tx=0).stop 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 -- 172.21.15.146:0/3402924197 shutdown_connections 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa0500745b0 0x7fa050076ac0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa06c15f920 0x7fa06c1f55e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa06c159960 0x7fa06c1565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 --2- 172.21.15.146:0/3402924197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa06c0b0220 0x7fa06c159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 -- 172.21.15.146:0/3402924197 >> 172.21.15.146:0/3402924197 conn(0x7fa06c0d1480 msgr2=0x7fa06c0b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 -- 172.21.15.146:0/3402924197 shutdown_connections 2024-04-25T15:20:13.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7fa0567fc640 1 -- 172.21.15.146:0/3402924197 wait complete. 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7f774e7fc640 1 -- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7754074580 msgr2=0x7f7754076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.648+0000 7f774e7fc640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7754074580 0x7f7754076a90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f775c006fd0 tx=0x7f775c008040 comp rx=0 tx=0).stop 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 -- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 msgr2=0x7f7768161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 0x7f7768161ac0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f77580098c0 tx=0x7f77580046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 -- 172.21.15.146:0/3850607986 shutdown_connections 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7754074580 0x7f7754076a90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77681654d0 0x7f776809ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77680c4ac0 0x7f7768161ac0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f77580098c0 tx=0x7f77580046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 --2- 172.21.15.146:0/3850607986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f776806c960 0x7f776815fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 -- 172.21.15.146:0/3850607986 >> 172.21.15.146:0/3850607986 conn(0x7f776809b2b0 msgr2=0x7f77680b6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 -- 172.21.15.146:0/3850607986 shutdown_connections 2024-04-25T15:20:13.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.649+0000 7f774e7fc640 1 -- 172.21.15.146:0/3850607986 wait complete. 2024-04-25T15:20:13.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-04-25T15:20:13.668 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-04-25T15:20:13.669 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.681+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4de006fba0 con 0x7f4de006c550 2024-04-25T15:20:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.682+0000 7f4ddcff9640 1 -- 172.21.15.146:0/3238908307 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4dd005e790 con 0x7f4de006c550 2024-04-25T15:20:13.683 INFO:teuthology.orchestra.run.smithi146.stdout:55834574859 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4dcc074580 msgr2=0x7f4dcc076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4dcc074580 0x7f4dcc076a90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4dd402a150 tx=0x7f4dd402e000 comp rx=0 tx=0).stop 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 msgr2=0x7f4de0146200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0146200 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000c780 tx=0x7f4dd000cc40 comp rx=0 tx=0).stop 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 shutdown_connections 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4dcc074580 0x7f4dcc076a90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de0149dc0 0x7f4de01ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de0147210 0x7f4de01fa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 --2- 172.21.15.146:0/3238908307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de006c550 0x7f4de0146200 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 >> 172.21.15.146:0/3238908307 conn(0x7f4de00cde40 msgr2=0x7f4de006cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 shutdown_connections 2024-04-25T15:20:13.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.685+0000 7f4de50cc640 1 -- 172.21.15.146:0/3238908307 wait complete. 2024-04-25T15:20:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.691+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb67815b4d0 con 0x7fb67806c8c0 2024-04-25T15:20:13.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.692+0000 7fb66effd640 1 -- 172.21.15.146:0/1842512368 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb670083a80 con 0x7fb67806c8c0 2024-04-25T15:20:13.692 INFO:teuthology.orchestra.run.smithi146.stdout:55834574859 2024-04-25T15:20:13.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb65c0748b0 msgr2=0x7fb65c076dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb65c0748b0 0x7fb65c076dc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb678158f00 tx=0x7fb668007e80 comp rx=0 tx=0).stop 2024-04-25T15:20:13.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 msgr2=0x7fb678156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 0x7fb678156cd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb670034040 tx=0x7fb670004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 shutdown_connections 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb65c0748b0 0x7fb65c076dc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6781618f0 0x7fb6780b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6780cc280 0x7fb6780b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 --2- 172.21.15.146:0/1842512368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb67806c8c0 0x7fb678156cd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 >> 172.21.15.146:0/1842512368 conn(0x7fb6780ce0c0 msgr2=0x7fb6781594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 shutdown_connections 2024-04-25T15:20:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.695+0000 7fb6805f3640 1 -- 172.21.15.146:0/1842512368 wait complete. 2024-04-25T15:20:13.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2024-04-25T15:20:13.701 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.708 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.5 2024-04-25T15:20:13.709 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.723+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0ea4005850 con 0x7f0ec4151d60 2024-04-25T15:20:13.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.723+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7feb2c005580 con 0x7feb4409ebe0 2024-04-25T15:20:13.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.724+0000 7feb3a7fc640 1 -- 172.21.15.146:0/822636139 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feb4416fbf0 con 0x7feb4409ebe0 2024-04-25T15:20:13.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.724+0000 7f0eab7fe640 1 -- 172.21.15.146:0/63265493 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0eb4062940 con 0x7f0ec4151d60 2024-04-25T15:20:13.725 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:13.725 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:13.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.725+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f351c005580 con 0x7f353809c520 2024-04-25T15:20:13.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.725+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7faee006a0e0 con 0x7faee006f6d0 2024-04-25T15:20:13.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.725+0000 7f3526ffd640 1 -- 172.21.15.146:0/3922514845 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3530010040 con 0x7f353809c520 2024-04-25T15:20:13.727 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:13.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.726+0000 7faedcff9640 1 -- 172.21.15.146:0/1084261431 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faed4068030 con 0x7faee006f6d0 2024-04-25T15:20:13.727 INFO:teuthology.orchestra.run.smithi146.stdout:55834574858 2024-04-25T15:20:13.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feb280745b0 msgr2=0x7feb28076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feb280745b0 0x7feb28076ac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feb44167630 tx=0x7feb40008040 comp rx=0 tx=0).stop 2024-04-25T15:20:13.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0eb00744b0 msgr2=0x7f0eb00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0eb00744b0 0x7f0eb00769c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc033000 tx=0x7f0ebc009000 comp rx=0 tx=0).stop 2024-04-25T15:20:13.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 msgr2=0x7feb44162950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 0x7feb44162950 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb3c001d10 tx=0x7feb3c0021d0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec4151d60 msgr2=0x7f0ec40ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec4151d60 0x7f0ec40ca880 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40cf9c0 tx=0x7f0eb4001e70 comp rx=0 tx=0).stop 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 shutdown_connections 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0eb00744b0 0x7f0eb00769c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ec4151d60 0x7f0ec40ca880 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ec4160f60 0x7f0ec4150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 --2- 172.21.15.146:0/63265493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ec406c550 0x7f0ec4150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 >> 172.21.15.146:0/63265493 conn(0x7f0ec409b2b0 msgr2=0x7f0ec415c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 shutdown_connections 2024-04-25T15:20:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feb280745b0 0x7feb28076ac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb4416f7e0 0x7feb440b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb44167220 0x7feb4416f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 --2- 172.21.15.146:0/822636139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4409ebe0 0x7feb44162950 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 >> 172.21.15.146:0/822636139 conn(0x7feb440ce0c0 msgr2=0x7feb4406c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f35180744b0 msgr2=0x7f35180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f35180744b0 0x7f35180769c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3528009560 tx=0x7f352802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 msgr2=0x7f353809c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 0x7f353809c930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f353000ee00 tx=0x7f353000c490 comp rx=0 tx=0).stop 2024-04-25T15:20:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 shutdown_connections 2024-04-25T15:20:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 shutdown_connections 2024-04-25T15:20:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7feb4bfce640 1 -- 172.21.15.146:0/822636139 wait complete. 2024-04-25T15:20:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f35180744b0 0x7f35180769c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 shutdown_connections 2024-04-25T15:20:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f353809c520 0x7f353809c930 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35381645c0 0x7f353809bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 --2- 172.21.15.146:0/3922514845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f353806cc60 0x7f35380ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 >> 172.21.15.146:0/3922514845 conn(0x7f35380ce0c0 msgr2=0x7f35380ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f0ec9546640 1 -- 172.21.15.146:0/63265493 wait complete. 2024-04-25T15:20:13.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 shutdown_connections 2024-04-25T15:20:13.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7f353cfce640 1 -- 172.21.15.146:0/3922514845 wait complete. 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faecc0745b0 msgr2=0x7faecc076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faecc0745b0 0x7faecc076ac0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faed0000f80 tx=0x7faed0009000 comp rx=0 tx=0).stop 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 msgr2=0x7faee014ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 0x7faee014ac40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faed400d520 tx=0x7faed400d9e0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 shutdown_connections 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faecc0745b0 0x7faecc076ac0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee00d21f0 0x7faee015ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee006f6d0 0x7faee014ac40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 --2- 172.21.15.146:0/1084261431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee006f2c0 0x7faee014a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 >> 172.21.15.146:0/1084261431 conn(0x7faee00b7530 msgr2=0x7faee00cd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 shutdown_connections 2024-04-25T15:20:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.728+0000 7faec67fc640 1 -- 172.21.15.146:0/1084261431 wait complete. 2024-04-25T15:20:13.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-04-25T15:20:13.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-25T15:20:13.744 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.744 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2024-04-25T15:20:13.749 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.4 2024-04-25T15:20:13.749 DEBUG:teuthology.parallel:result is None 2024-04-25T15:20:13.750 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:20:13.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:20:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.895+0000 7fcd55884640 1 Processor -- start 2024-04-25T15:20:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.895+0000 7fcd55884640 1 -- start start 2024-04-25T15:20:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd55884640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcd500d4f60 0x7fcd500d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd55884640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 0x7fcd500d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd55884640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd500b9200 con 0x7fcd50163d80 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd55884640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcd5006c550 con 0x7fcd500d4f60 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd55884640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5006c830 con 0x7fcd500d0020 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcd50163d80 0x7fcd50164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40648/0 (socket says 172.21.15.146:40648) 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4f7fe640 1 -- 172.21.15.146:0/3679209342 learned_addr learned my addr 172.21.15.146:0/3679209342 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4effd640 1 --2- 172.21.15.146:0/3679209342 >> v2:172.21.15.177:3300/0 conn(0x7fcd500d4f60 0x7fcd500d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4e7fc640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 0x7fcd500d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433410528 0 0) 0x7fcd48001110 con 0x7fcd50163d80 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd38003520 con 0x7fcd50163d80 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3489125392 0 0) 0x7fcd38003520 con 0x7fcd50163d80 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 msgr2=0x7fcd500d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 0x7fcd500d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 >> v2:172.21.15.177:3300/0 conn(0x7fcd500d4f60 msgr2=0x7fcd500d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 --2- 172.21.15.146:0/3679209342 >> v2:172.21.15.177:3300/0 conn(0x7fcd500d4f60 0x7fcd500d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd500d0970 con 0x7fcd50163d80 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fcd48001840 con 0x7fcd50163d80 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 >> v1:172.21.15.146:6789/0 conn(0x7fcd50163d80 legacy=0x7fcd50164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 0x7fcd380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd38003f30 0x7fcd38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38004d40 0x7fcd38005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5006c550 con 0x7fcd38003f30 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5006c830 con 0x7fcd38004d40 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.897+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcd3800bb50 con 0x7fcd500d0020 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd38003f30 0x7fcd38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4effd640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 0x7fcd380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4f7fe640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38004d40 0x7fcd38005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 -- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 msgr2=0x7fcd380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd500d0020 0x7fcd380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 -- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38004d40 msgr2=0x7fcd38005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38004d40 0x7fcd38005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 -- 172.21.15.146:0/3679209342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd38003520 con 0x7fcd38003f30 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.898+0000 7fcd4e7fc640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd38003f30 0x7fcd38004800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009a90 tx=0x7fcd38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1ec2ba748d1a40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd44036070 con 0x7fcd38003f30 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd44031400 con 0x7fcd38003f30 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd4dffb640 1 -- 172.21.15.146:0/3679209342 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd44009690 con 0x7fcd38003f30 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd55884640 1 -- 172.21.15.146:0/3679209342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd38003f30 msgr2=0x7fcd38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd55884640 1 --2- 172.21.15.146:0/3679209342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd38003f30 0x7fcd38004800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009a90 tx=0x7fcd38005830 comp rx=0 tx=0).stop 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd4f7fe640 1 -- 172.21.15.146:0/3679209342 reap_dead start 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd55884640 1 -- 172.21.15.146:0/3679209342 shutdown_connections 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd55884640 1 -- 172.21.15.146:0/3679209342 >> 172.21.15.146:0/3679209342 conn(0x7fcd500d2450 msgr2=0x7fcd500d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd55884640 1 -- 172.21.15.146:0/3679209342 shutdown_connections 2024-04-25T15:20:13.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.899+0000 7fcd55884640 1 -- 172.21.15.146:0/3679209342 wait complete. 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.901+0000 7fcd55884640 1 Processor -- start 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.901+0000 7fcd55884640 1 -- start start 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.902+0000 7fcd55884640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38003f30 0x7fcd5014e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5014b380 0x7fcd50161a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 0x7fcd50161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcd500d15f0 con 0x7fcd5014b790 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd500d1070 con 0x7fcd38003f30 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcd500b5150 con 0x7fcd5014b380 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 0x7fcd50161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 0x7fcd50161fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60312/0 (socket says 172.21.15.146:60312) 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 -- 172.21.15.146:0/563639097 learned_addr learned my addr 172.21.15.146:0/563639097 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4e7fc640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5014b380 0x7fcd50161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4effd640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38003f30 0x7fcd5014e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 -- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5014b380 msgr2=0x7fcd50161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5014b380 0x7fcd50161a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 -- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38003f30 msgr2=0x7fcd5014e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38003f30 0x7fcd5014e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 -- 172.21.15.146:0/563639097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd501ea1a0 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd4f7fe640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 0x7fcd50161fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcd500d3e30 tx=0x7fcd48007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4801a070 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd4800f940 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd501ea480 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48001180 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.903+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5014ec90 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.904+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd5006a0e0 con 0x7fcd5014b790 2024-04-25T15:20:13.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.905+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcd4801d070 con 0x7fcd5014b790 2024-04-25T15:20:13.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.905+0000 7fcd37fff640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd3c0744e0 0x7fcd3c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:13.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.906+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcd48094990 con 0x7fcd5014b790 2024-04-25T15:20:13.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.906+0000 7fcd4effd640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd3c0744e0 0x7fcd3c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:13.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.906+0000 7fcd4effd640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd3c0744e0 0x7fcd3c0769f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcd500b4880 tx=0x7fcd4000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:13.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:13.910+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd500d0970 con 0x7fcd5014b790 2024-04-25T15:20:14.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.084+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd500b5f90 con 0x7fcd3c0744e0 2024-04-25T15:20:14.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.092+0000 7fcd37fff640 1 -- 172.21.15.146:0/563639097 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332522 (secure 0 0 0) 0x7fcd500b5f90 con 0x7fcd3c0744e0 2024-04-25T15:20:14.098 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:14.100 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:20:14.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd3c0744e0 msgr2=0x7fcd3c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd3c0744e0 0x7fcd3c0769f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcd500b4880 tx=0x7fcd4000b040 comp rx=0 tx=0).stop 2024-04-25T15:20:14.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 msgr2=0x7fcd50161fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 0x7fcd50161fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcd500d3e30 tx=0x7fcd48007720 comp rx=0 tx=0).stop 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 shutdown_connections 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd3c0744e0 0x7fcd3c0769f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5014b790 0x7fcd50161fd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5014b380 0x7fcd50161a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 --2- 172.21.15.146:0/563639097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd38003f30 0x7fcd5014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 >> 172.21.15.146:0/563639097 conn(0x7fcd500d2450 msgr2=0x7fcd500ba7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 shutdown_connections 2024-04-25T15:20:14.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.096+0000 7fcd55884640 1 -- 172.21.15.146:0/563639097 wait complete. 2024-04-25T15:20:14.109 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-25T15:20:13.993556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003220"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489335+0000","last_change":"2024-04-25T15:19:45.467889+0000","last_active":"2024-04-25T15:19:52.489335+0000","last_peered":"2024-04-25T15:19:52.489335+0000","last_clean":"2024-04-25T15:19:52.489335+0000","last_became_active":"2024-04-25T15:19:45.467565+0000","last_became_peered":"2024-04-25T15:19:45.467565+0000","last_unstale":"2024-04-25T15:19:52.489335+0000","last_undegraded":"2024-04-25T15:19:52.489335+0000","last_fullsized":"2024-04-25T15:19:52.489335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020856+0000","last_change":"2024-04-25T15:19:46.476186+0000","last_active":"2024-04-25T15:19:54.020856+0000","last_peered":"2024-04-25T15:19:54.020856+0000","last_clean":"2024-04-25T15:19:54.020856+0000","last_became_active":"2024-04-25T15:19:46.475865+0000","last_became_peered":"2024-04-25T15:19:46.475865+0000","last_unstale":"2024-04-25T15:19:54.020856+0000","last_undegraded":"2024-04-25T15:19:54.020856+0000","last_fullsized":"2024-04-25T15:19:54.020856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020377+0000","last_change":"2024-04-25T15:19:51.102406+0000","last_active":"2024-04-25T15:19:54.020377+0000","last_peered":"2024-04-25T15:19:54.020377+0000","last_clean":"2024-04-25T15:19:54.020377+0000","last_became_active":"2024-04-25T15:19:51.101849+0000","last_became_peered":"2024-04-25T15:19:51.101849+0000","last_unstale":"2024-04-25T15:19:54.020377+0000","last_undegraded":"2024-04-25T15:19:54.020377+0000","last_fullsized":"2024-04-25T15:19:54.020377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:58.329289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020474+0000","last_change":"2024-04-25T15:19:45.461077+0000","last_active":"2024-04-25T15:19:54.020474+0000","last_peered":"2024-04-25T15:19:54.020474+0000","last_clean":"2024-04-25T15:19:54.020474+0000","last_became_active":"2024-04-25T15:19:45.460662+0000","last_became_peered":"2024-04-25T15:19:45.460662+0000","last_unstale":"2024-04-25T15:19:54.020474+0000","last_undegraded":"2024-04-25T15:19:54.020474+0000","last_fullsized":"2024-04-25T15:19:54.020474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489205+0000","last_change":"2024-04-25T15:19:46.486683+0000","last_active":"2024-04-25T15:19:52.489205+0000","last_peered":"2024-04-25T15:19:52.489205+0000","last_clean":"2024-04-25T15:19:52.489205+0000","last_became_active":"2024-04-25T15:19:46.486203+0000","last_became_peered":"2024-04-25T15:19:46.486203+0000","last_unstale":"2024-04-25T15:19:52.489205+0000","last_undegraded":"2024-04-25T15:19:52.489205+0000","last_fullsized":"2024-04-25T15:19:52.489205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490288+0000","last_change":"2024-04-25T15:19:50.510692+0000","last_active":"2024-04-25T15:19:52.490288+0000","last_peered":"2024-04-25T15:19:52.490288+0000","last_clean":"2024-04-25T15:19:52.490288+0000","last_became_active":"2024-04-25T15:19:50.510156+0000","last_became_peered":"2024-04-25T15:19:50.510156+0000","last_unstale":"2024-04-25T15:19:52.490288+0000","last_undegraded":"2024-04-25T15:19:52.490288+0000","last_fullsized":"2024-04-25T15:19:52.490288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396965+0000","last_change":"2024-04-25T15:19:45.471053+0000","last_active":"2024-04-25T15:19:53.396965+0000","last_peered":"2024-04-25T15:19:53.396965+0000","last_clean":"2024-04-25T15:19:53.396965+0000","last_became_active":"2024-04-25T15:19:45.470704+0000","last_became_peered":"2024-04-25T15:19:45.470704+0000","last_unstale":"2024-04-25T15:19:53.396965+0000","last_undegraded":"2024-04-25T15:19:53.396965+0000","last_fullsized":"2024-04-25T15:19:53.396965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:55.012973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491359+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.491359+0000","last_peered":"2024-04-25T15:19:52.491359+0000","last_clean":"2024-04-25T15:19:52.491359+0000","last_became_active":"2024-04-25T15:19:46.482181+0000","last_became_peered":"2024-04-25T15:19:46.482181+0000","last_unstale":"2024-04-25T15:19:52.491359+0000","last_undegraded":"2024-04-25T15:19:52.491359+0000","last_fullsized":"2024-04-25T15:19:52.491359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492195+0000","last_change":"2024-04-25T15:19:50.515950+0000","last_active":"2024-04-25T15:19:52.492195+0000","last_peered":"2024-04-25T15:19:52.492195+0000","last_clean":"2024-04-25T15:19:52.492195+0000","last_became_active":"2024-04-25T15:19:50.515486+0000","last_became_peered":"2024-04-25T15:19:50.515486+0000","last_unstale":"2024-04-25T15:19:52.492195+0000","last_undegraded":"2024-04-25T15:19:52.492195+0000","last_fullsized":"2024-04-25T15:19:52.492195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396633+0000","last_change":"2024-04-25T15:19:45.472051+0000","last_active":"2024-04-25T15:19:53.396633+0000","last_peered":"2024-04-25T15:19:53.396633+0000","last_clean":"2024-04-25T15:19:53.396633+0000","last_became_active":"2024-04-25T15:19:45.471710+0000","last_became_peered":"2024-04-25T15:19:45.471710+0000","last_unstale":"2024-04-25T15:19:53.396633+0000","last_undegraded":"2024-04-25T15:19:53.396633+0000","last_fullsized":"2024-04-25T15:19:53.396633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491729+0000","last_change":"2024-04-25T15:19:46.483085+0000","last_active":"2024-04-25T15:19:52.491729+0000","last_peered":"2024-04-25T15:19:52.491729+0000","last_clean":"2024-04-25T15:19:52.491729+0000","last_became_active":"2024-04-25T15:19:46.482749+0000","last_became_peered":"2024-04-25T15:19:46.482749+0000","last_unstale":"2024-04-25T15:19:52.491729+0000","last_undegraded":"2024-04-25T15:19:52.491729+0000","last_fullsized":"2024-04-25T15:19:52.491729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490380+0000","last_change":"2024-04-25T15:19:50.508661+0000","last_active":"2024-04-25T15:19:52.490380+0000","last_peered":"2024-04-25T15:19:52.490380+0000","last_clean":"2024-04-25T15:19:52.490380+0000","last_became_active":"2024-04-25T15:19:50.508116+0000","last_became_peered":"2024-04-25T15:19:50.508116+0000","last_unstale":"2024-04-25T15:19:52.490380+0000","last_undegraded":"2024-04-25T15:19:52.490380+0000","last_fullsized":"2024-04-25T15:19:52.490380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018240+0000","last_change":"2024-04-25T15:19:46.477569+0000","last_active":"2024-04-25T15:19:54.018240+0000","last_peered":"2024-04-25T15:19:54.018240+0000","last_clean":"2024-04-25T15:19:54.018240+0000","last_became_active":"2024-04-25T15:19:46.477188+0000","last_became_peered":"2024-04-25T15:19:46.477188+0000","last_unstale":"2024-04-25T15:19:54.018240+0000","last_undegraded":"2024-04-25T15:19:54.018240+0000","last_fullsized":"2024-04-25T15:19:54.018240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396748+0000","last_change":"2024-04-25T15:19:45.469111+0000","last_active":"2024-04-25T15:19:53.396748+0000","last_peered":"2024-04-25T15:19:53.396748+0000","last_clean":"2024-04-25T15:19:53.396748+0000","last_became_active":"2024-04-25T15:19:45.468780+0000","last_became_peered":"2024-04-25T15:19:45.468780+0000","last_unstale":"2024-04-25T15:19:53.396748+0000","last_undegraded":"2024-04-25T15:19:53.396748+0000","last_fullsized":"2024-04-25T15:19:53.396748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491203+0000","last_change":"2024-04-25T15:19:50.513416+0000","last_active":"2024-04-25T15:19:52.491203+0000","last_peered":"2024-04-25T15:19:52.491203+0000","last_clean":"2024-04-25T15:19:52.491203+0000","last_became_active":"2024-04-25T15:19:50.513024+0000","last_became_peered":"2024-04-25T15:19:50.513024+0000","last_unstale":"2024-04-25T15:19:52.491203+0000","last_undegraded":"2024-04-25T15:19:52.491203+0000","last_fullsized":"2024-04-25T15:19:52.491203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:20.334921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020296+0000","last_change":"2024-04-25T15:19:46.472167+0000","last_active":"2024-04-25T15:19:54.020296+0000","last_peered":"2024-04-25T15:19:54.020296+0000","last_clean":"2024-04-25T15:19:54.020296+0000","last_became_active":"2024-04-25T15:19:46.471839+0000","last_became_peered":"2024-04-25T15:19:46.471839+0000","last_unstale":"2024-04-25T15:19:54.020296+0000","last_undegraded":"2024-04-25T15:19:54.020296+0000","last_fullsized":"2024-04-25T15:19:54.020296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809709+0000","last_change":"2024-04-25T15:19:45.467435+0000","last_active":"2024-04-25T15:19:53.809709+0000","last_peered":"2024-04-25T15:19:53.809709+0000","last_clean":"2024-04-25T15:19:53.809709+0000","last_became_active":"2024-04-25T15:19:45.466960+0000","last_became_peered":"2024-04-25T15:19:45.466960+0000","last_unstale":"2024-04-25T15:19:53.809709+0000","last_undegraded":"2024-04-25T15:19:53.809709+0000","last_fullsized":"2024-04-25T15:19:53.809709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491020+0000","last_change":"2024-04-25T15:19:50.515801+0000","last_active":"2024-04-25T15:19:52.491020+0000","last_peered":"2024-04-25T15:19:52.491020+0000","last_clean":"2024-04-25T15:19:52.491020+0000","last_became_active":"2024-04-25T15:19:50.515340+0000","last_became_peered":"2024-04-25T15:19:50.515340+0000","last_unstale":"2024-04-25T15:19:52.491020+0000","last_undegraded":"2024-04-25T15:19:52.491020+0000","last_fullsized":"2024-04-25T15:19:52.491020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:39.634911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020955+0000","last_change":"2024-04-25T15:19:46.470328+0000","last_active":"2024-04-25T15:19:54.020955+0000","last_peered":"2024-04-25T15:19:54.020955+0000","last_clean":"2024-04-25T15:19:54.020955+0000","last_became_active":"2024-04-25T15:19:46.469972+0000","last_became_peered":"2024-04-25T15:19:46.469972+0000","last_unstale":"2024-04-25T15:19:54.020955+0000","last_undegraded":"2024-04-25T15:19:54.020955+0000","last_fullsized":"2024-04-25T15:19:54.020955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.808963+0000","last_change":"2024-04-25T15:19:45.468754+0000","last_active":"2024-04-25T15:19:53.808963+0000","last_peered":"2024-04-25T15:19:53.808963+0000","last_clean":"2024-04-25T15:19:53.808963+0000","last_became_active":"2024-04-25T15:19:45.468374+0000","last_became_peered":"2024-04-25T15:19:45.468374+0000","last_unstale":"2024-04-25T15:19:53.808963+0000","last_undegraded":"2024-04-25T15:19:53.808963+0000","last_fullsized":"2024-04-25T15:19:53.808963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810131+0000","last_change":"2024-04-25T15:19:51.101484+0000","last_active":"2024-04-25T15:19:53.810131+0000","last_peered":"2024-04-25T15:19:53.810131+0000","last_clean":"2024-04-25T15:19:53.810131+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:53.810131+0000","last_undegraded":"2024-04-25T15:19:53.810131+0000","last_fullsized":"2024-04-25T15:19:53.810131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:57.500581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491979+0000","last_change":"2024-04-25T15:19:46.458762+0000","last_active":"2024-04-25T15:19:52.491979+0000","last_peered":"2024-04-25T15:19:52.491979+0000","last_clean":"2024-04-25T15:19:52.491979+0000","last_became_active":"2024-04-25T15:19:46.458438+0000","last_became_peered":"2024-04-25T15:19:46.458438+0000","last_unstale":"2024-04-25T15:19:52.491979+0000","last_undegraded":"2024-04-25T15:19:52.491979+0000","last_fullsized":"2024-04-25T15:19:52.491979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.402061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396261+0000","last_change":"2024-04-25T15:19:45.471672+0000","last_active":"2024-04-25T15:19:53.396261+0000","last_peered":"2024-04-25T15:19:53.396261+0000","last_clean":"2024-04-25T15:19:53.396261+0000","last_became_active":"2024-04-25T15:19:45.471095+0000","last_became_peered":"2024-04-25T15:19:45.471095+0000","last_unstale":"2024-04-25T15:19:53.396261+0000","last_undegraded":"2024-04-25T15:19:53.396261+0000","last_fullsized":"2024-04-25T15:19:53.396261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490630+0000","last_change":"2024-04-25T15:19:50.511923+0000","last_active":"2024-04-25T15:19:52.490630+0000","last_peered":"2024-04-25T15:19:52.490630+0000","last_clean":"2024-04-25T15:19:52.490630+0000","last_became_active":"2024-04-25T15:19:50.511350+0000","last_became_peered":"2024-04-25T15:19:50.511350+0000","last_unstale":"2024-04-25T15:19:52.490630+0000","last_undegraded":"2024-04-25T15:19:52.490630+0000","last_fullsized":"2024-04-25T15:19:52.490630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489343+0000","last_change":"2024-04-25T15:19:46.472562+0000","last_active":"2024-04-25T15:19:52.489343+0000","last_peered":"2024-04-25T15:19:52.489343+0000","last_clean":"2024-04-25T15:19:52.489343+0000","last_became_active":"2024-04-25T15:19:46.471397+0000","last_became_peered":"2024-04-25T15:19:46.471397+0000","last_unstale":"2024-04-25T15:19:52.489343+0000","last_undegraded":"2024-04-25T15:19:52.489343+0000","last_fullsized":"2024-04-25T15:19:52.489343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017764+0000","last_change":"2024-04-25T15:19:45.467859+0000","last_active":"2024-04-25T15:19:54.017764+0000","last_peered":"2024-04-25T15:19:54.017764+0000","last_clean":"2024-04-25T15:19:54.017764+0000","last_became_active":"2024-04-25T15:19:45.467461+0000","last_became_peered":"2024-04-25T15:19:45.467461+0000","last_unstale":"2024-04-25T15:19:54.017764+0000","last_undegraded":"2024-04-25T15:19:54.017764+0000","last_fullsized":"2024-04-25T15:19:54.017764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491639+0000","last_change":"2024-04-25T15:19:51.101973+0000","last_active":"2024-04-25T15:19:52.491639+0000","last_peered":"2024-04-25T15:19:52.491639+0000","last_clean":"2024-04-25T15:19:52.491639+0000","last_became_active":"2024-04-25T15:19:51.101046+0000","last_became_peered":"2024-04-25T15:19:51.101046+0000","last_unstale":"2024-04-25T15:19:52.491639+0000","last_undegraded":"2024-04-25T15:19:52.491639+0000","last_fullsized":"2024-04-25T15:19:52.491639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020585+0000","last_change":"2024-04-25T15:19:46.481579+0000","last_active":"2024-04-25T15:19:54.020585+0000","last_peered":"2024-04-25T15:19:54.020585+0000","last_clean":"2024-04-25T15:19:54.020585+0000","last_became_active":"2024-04-25T15:19:46.481265+0000","last_became_peered":"2024-04-25T15:19:46.481265+0000","last_unstale":"2024-04-25T15:19:54.020585+0000","last_undegraded":"2024-04-25T15:19:54.020585+0000","last_fullsized":"2024-04-25T15:19:54.020585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810014+0000","last_change":"2024-04-25T15:19:45.464607+0000","last_active":"2024-04-25T15:19:53.810014+0000","last_peered":"2024-04-25T15:19:53.810014+0000","last_clean":"2024-04-25T15:19:53.810014+0000","last_became_active":"2024-04-25T15:19:45.464278+0000","last_became_peered":"2024-04-25T15:19:45.464278+0000","last_unstale":"2024-04-25T15:19:53.810014+0000","last_undegraded":"2024-04-25T15:19:53.810014+0000","last_fullsized":"2024-04-25T15:19:53.810014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020107+0000","last_change":"2024-04-25T15:19:51.101501+0000","last_active":"2024-04-25T15:19:54.020107+0000","last_peered":"2024-04-25T15:19:54.020107+0000","last_clean":"2024-04-25T15:19:54.020107+0000","last_became_active":"2024-04-25T15:19:51.101130+0000","last_became_peered":"2024-04-25T15:19:51.101130+0000","last_unstale":"2024-04-25T15:19:54.020107+0000","last_undegraded":"2024-04-25T15:19:54.020107+0000","last_fullsized":"2024-04-25T15:19:54.020107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809775+0000","last_change":"2024-04-25T15:19:46.469600+0000","last_active":"2024-04-25T15:19:53.809775+0000","last_peered":"2024-04-25T15:19:53.809775+0000","last_clean":"2024-04-25T15:19:53.809775+0000","last_became_active":"2024-04-25T15:19:46.469254+0000","last_became_peered":"2024-04-25T15:19:46.469254+0000","last_unstale":"2024-04-25T15:19:53.809775+0000","last_undegraded":"2024-04-25T15:19:53.809775+0000","last_fullsized":"2024-04-25T15:19:53.809775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397107+0000","last_change":"2024-04-25T15:19:45.471728+0000","last_active":"2024-04-25T15:19:53.397107+0000","last_peered":"2024-04-25T15:19:53.397107+0000","last_clean":"2024-04-25T15:19:53.397107+0000","last_became_active":"2024-04-25T15:19:45.471426+0000","last_became_peered":"2024-04-25T15:19:45.471426+0000","last_unstale":"2024-04-25T15:19:53.397107+0000","last_undegraded":"2024-04-25T15:19:53.397107+0000","last_fullsized":"2024-04-25T15:19:53.397107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:39:45.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488880+0000","last_change":"2024-04-25T15:19:51.101520+0000","last_active":"2024-04-25T15:19:52.488880+0000","last_peered":"2024-04-25T15:19:52.488880+0000","last_clean":"2024-04-25T15:19:52.488880+0000","last_became_active":"2024-04-25T15:19:51.100989+0000","last_became_peered":"2024-04-25T15:19:51.100989+0000","last_unstale":"2024-04-25T15:19:52.488880+0000","last_undegraded":"2024-04-25T15:19:52.488880+0000","last_fullsized":"2024-04-25T15:19:52.488880+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491850+0000","last_change":"2024-04-25T15:19:46.482591+0000","last_active":"2024-04-25T15:19:52.491850+0000","last_peered":"2024-04-25T15:19:52.491850+0000","last_clean":"2024-04-25T15:19:52.491850+0000","last_became_active":"2024-04-25T15:19:46.482265+0000","last_became_peered":"2024-04-25T15:19:46.482265+0000","last_unstale":"2024-04-25T15:19:52.491850+0000","last_undegraded":"2024-04-25T15:19:52.491850+0000","last_fullsized":"2024-04-25T15:19:52.491850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809000+0000","last_change":"2024-04-25T15:19:45.470984+0000","last_active":"2024-04-25T15:19:53.809000+0000","last_peered":"2024-04-25T15:19:53.809000+0000","last_clean":"2024-04-25T15:19:53.809000+0000","last_became_active":"2024-04-25T15:19:45.470665+0000","last_became_peered":"2024-04-25T15:19:45.470665+0000","last_unstale":"2024-04-25T15:19:53.809000+0000","last_undegraded":"2024-04-25T15:19:53.809000+0000","last_fullsized":"2024-04-25T15:19:53.809000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809497+0000","last_change":"2024-04-25T15:19:50.509138+0000","last_active":"2024-04-25T15:19:53.809497+0000","last_peered":"2024-04-25T15:19:53.809497+0000","last_clean":"2024-04-25T15:19:53.809497+0000","last_became_active":"2024-04-25T15:19:50.508713+0000","last_became_peered":"2024-04-25T15:19:50.508713+0000","last_unstale":"2024-04-25T15:19:53.809497+0000","last_undegraded":"2024-04-25T15:19:53.809497+0000","last_fullsized":"2024-04-25T15:19:53.809497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492266+0000","last_change":"2024-04-25T15:19:46.459065+0000","last_active":"2024-04-25T15:19:52.492266+0000","last_peered":"2024-04-25T15:19:52.492266+0000","last_clean":"2024-04-25T15:19:52.492266+0000","last_became_active":"2024-04-25T15:19:46.458708+0000","last_became_peered":"2024-04-25T15:19:46.458708+0000","last_unstale":"2024-04-25T15:19:52.492266+0000","last_undegraded":"2024-04-25T15:19:52.492266+0000","last_fullsized":"2024-04-25T15:19:52.492266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:16.345566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491129+0000","last_change":"2024-04-25T15:19:45.464370+0000","last_active":"2024-04-25T15:19:52.491129+0000","last_peered":"2024-04-25T15:19:52.491129+0000","last_clean":"2024-04-25T15:19:52.491129+0000","last_became_active":"2024-04-25T15:19:45.464041+0000","last_became_peered":"2024-04-25T15:19:45.464041+0000","last_unstale":"2024-04-25T15:19:52.491129+0000","last_undegraded":"2024-04-25T15:19:52.491129+0000","last_fullsized":"2024-04-25T15:19:52.491129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809793+0000","last_change":"2024-04-25T15:19:50.516748+0000","last_active":"2024-04-25T15:19:53.809793+0000","last_peered":"2024-04-25T15:19:53.809793+0000","last_clean":"2024-04-25T15:19:53.809793+0000","last_became_active":"2024-04-25T15:19:50.515480+0000","last_became_peered":"2024-04-25T15:19:50.515480+0000","last_unstale":"2024-04-25T15:19:53.809793+0000","last_undegraded":"2024-04-25T15:19:53.809793+0000","last_fullsized":"2024-04-25T15:19:53.809793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:54.507021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809413+0000","last_change":"2024-04-25T15:19:46.469618+0000","last_active":"2024-04-25T15:19:53.809413+0000","last_peered":"2024-04-25T15:19:53.809413+0000","last_clean":"2024-04-25T15:19:53.809413+0000","last_became_active":"2024-04-25T15:19:46.469289+0000","last_became_peered":"2024-04-25T15:19:46.469289+0000","last_unstale":"2024-04-25T15:19:53.809413+0000","last_undegraded":"2024-04-25T15:19:53.809413+0000","last_fullsized":"2024-04-25T15:19:53.809413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020301+0000","last_change":"2024-04-25T15:19:45.462951+0000","last_active":"2024-04-25T15:19:54.020301+0000","last_peered":"2024-04-25T15:19:54.020301+0000","last_clean":"2024-04-25T15:19:54.020301+0000","last_became_active":"2024-04-25T15:19:45.462424+0000","last_became_peered":"2024-04-25T15:19:45.462424+0000","last_unstale":"2024-04-25T15:19:54.020301+0000","last_undegraded":"2024-04-25T15:19:54.020301+0000","last_fullsized":"2024-04-25T15:19:54.020301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810177+0000","last_change":"2024-04-25T15:19:50.519794+0000","last_active":"2024-04-25T15:19:53.810177+0000","last_peered":"2024-04-25T15:19:53.810177+0000","last_clean":"2024-04-25T15:19:53.810177+0000","last_became_active":"2024-04-25T15:19:50.519278+0000","last_became_peered":"2024-04-25T15:19:50.519278+0000","last_unstale":"2024-04-25T15:19:53.810177+0000","last_undegraded":"2024-04-25T15:19:53.810177+0000","last_fullsized":"2024-04-25T15:19:53.810177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488686+0000","last_change":"2024-04-25T15:19:46.459839+0000","last_active":"2024-04-25T15:19:52.488686+0000","last_peered":"2024-04-25T15:19:52.488686+0000","last_clean":"2024-04-25T15:19:52.488686+0000","last_became_active":"2024-04-25T15:19:46.459401+0000","last_became_peered":"2024-04-25T15:19:46.459401+0000","last_unstale":"2024-04-25T15:19:52.488686+0000","last_undegraded":"2024-04-25T15:19:52.488686+0000","last_fullsized":"2024-04-25T15:19:52.488686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:46.303899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396291+0000","last_change":"2024-04-25T15:19:45.471412+0000","last_active":"2024-04-25T15:19:53.396291+0000","last_peered":"2024-04-25T15:19:53.396291+0000","last_clean":"2024-04-25T15:19:53.396291+0000","last_became_active":"2024-04-25T15:19:45.470632+0000","last_became_peered":"2024-04-25T15:19:45.470632+0000","last_unstale":"2024-04-25T15:19:53.396291+0000","last_undegraded":"2024-04-25T15:19:53.396291+0000","last_fullsized":"2024-04-25T15:19:53.396291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489312+0000","last_change":"2024-04-25T15:19:51.101653+0000","last_active":"2024-04-25T15:19:52.489312+0000","last_peered":"2024-04-25T15:19:52.489312+0000","last_clean":"2024-04-25T15:19:52.489312+0000","last_became_active":"2024-04-25T15:19:51.101246+0000","last_became_peered":"2024-04-25T15:19:51.101246+0000","last_unstale":"2024-04-25T15:19:52.489312+0000","last_undegraded":"2024-04-25T15:19:52.489312+0000","last_fullsized":"2024-04-25T15:19:52.489312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020754+0000","last_change":"2024-04-25T15:19:46.488866+0000","last_active":"2024-04-25T15:19:54.020754+0000","last_peered":"2024-04-25T15:19:54.020754+0000","last_clean":"2024-04-25T15:19:54.020754+0000","last_became_active":"2024-04-25T15:19:46.488494+0000","last_became_peered":"2024-04-25T15:19:46.488494+0000","last_unstale":"2024-04-25T15:19:54.020754+0000","last_undegraded":"2024-04-25T15:19:54.020754+0000","last_fullsized":"2024-04-25T15:19:54.020754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490534+0000","last_change":"2024-04-25T15:19:45.459379+0000","last_active":"2024-04-25T15:19:52.490534+0000","last_peered":"2024-04-25T15:19:52.490534+0000","last_clean":"2024-04-25T15:19:52.490534+0000","last_became_active":"2024-04-25T15:19:45.458985+0000","last_became_peered":"2024-04-25T15:19:45.458985+0000","last_unstale":"2024-04-25T15:19:52.490534+0000","last_undegraded":"2024-04-25T15:19:52.490534+0000","last_fullsized":"2024-04-25T15:19:52.490534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396508+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:19:53.396508+0000","last_peered":"2024-04-25T15:19:53.396508+0000","last_clean":"2024-04-25T15:19:53.396508+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:19:53.396508+0000","last_undegraded":"2024-04-25T15:19:53.396508+0000","last_fullsized":"2024-04-25T15:19:53.396508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018620+0000","last_change":"2024-04-25T15:19:51.102362+0000","last_active":"2024-04-25T15:19:54.018620+0000","last_peered":"2024-04-25T15:19:54.018620+0000","last_clean":"2024-04-25T15:19:54.018620+0000","last_became_active":"2024-04-25T15:19:51.101137+0000","last_became_peered":"2024-04-25T15:19:51.101137+0000","last_unstale":"2024-04-25T15:19:54.018620+0000","last_undegraded":"2024-04-25T15:19:54.018620+0000","last_fullsized":"2024-04-25T15:19:54.018620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396808+0000","last_change":"2024-04-25T15:19:46.486533+0000","last_active":"2024-04-25T15:19:53.396808+0000","last_peered":"2024-04-25T15:19:53.396808+0000","last_clean":"2024-04-25T15:19:53.396808+0000","last_became_active":"2024-04-25T15:19:46.485775+0000","last_became_peered":"2024-04-25T15:19:46.485775+0000","last_unstale":"2024-04-25T15:19:53.396808+0000","last_undegraded":"2024-04-25T15:19:53.396808+0000","last_fullsized":"2024-04-25T15:19:53.396808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019912+0000","last_change":"2024-04-25T15:19:45.467231+0000","last_active":"2024-04-25T15:19:54.019912+0000","last_peered":"2024-04-25T15:19:54.019912+0000","last_clean":"2024-04-25T15:19:54.019912+0000","last_became_active":"2024-04-25T15:19:45.466874+0000","last_became_peered":"2024-04-25T15:19:45.466874+0000","last_unstale":"2024-04-25T15:19:54.019912+0000","last_undegraded":"2024-04-25T15:19:54.019912+0000","last_fullsized":"2024-04-25T15:19:54.019912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018302+0000","last_change":"2024-04-25T15:19:51.101630+0000","last_active":"2024-04-25T15:19:54.018302+0000","last_peered":"2024-04-25T15:19:54.018302+0000","last_clean":"2024-04-25T15:19:54.018302+0000","last_became_active":"2024-04-25T15:19:51.101121+0000","last_became_peered":"2024-04-25T15:19:51.101121+0000","last_unstale":"2024-04-25T15:19:54.018302+0000","last_undegraded":"2024-04-25T15:19:54.018302+0000","last_fullsized":"2024-04-25T15:19:54.018302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491492+0000","last_change":"2024-04-25T15:19:46.483183+0000","last_active":"2024-04-25T15:19:52.491492+0000","last_peered":"2024-04-25T15:19:52.491492+0000","last_clean":"2024-04-25T15:19:52.491492+0000","last_became_active":"2024-04-25T15:19:46.482618+0000","last_became_peered":"2024-04-25T15:19:46.482618+0000","last_unstale":"2024-04-25T15:19:52.491492+0000","last_undegraded":"2024-04-25T15:19:52.491492+0000","last_fullsized":"2024-04-25T15:19:52.491492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809024+0000","last_change":"2024-04-25T15:19:45.469018+0000","last_active":"2024-04-25T15:19:53.809024+0000","last_peered":"2024-04-25T15:19:53.809024+0000","last_clean":"2024-04-25T15:19:53.809024+0000","last_became_active":"2024-04-25T15:19:45.468712+0000","last_became_peered":"2024-04-25T15:19:45.468712+0000","last_unstale":"2024-04-25T15:19:53.809024+0000","last_undegraded":"2024-04-25T15:19:53.809024+0000","last_fullsized":"2024-04-25T15:19:53.809024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489037+0000","last_change":"2024-04-25T15:19:50.513573+0000","last_active":"2024-04-25T15:19:52.489037+0000","last_peered":"2024-04-25T15:19:52.489037+0000","last_clean":"2024-04-25T15:19:52.489037+0000","last_became_active":"2024-04-25T15:19:50.513018+0000","last_became_peered":"2024-04-25T15:19:50.513018+0000","last_unstale":"2024-04-25T15:19:52.489037+0000","last_undegraded":"2024-04-25T15:19:52.489037+0000","last_fullsized":"2024-04-25T15:19:52.489037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397172+0000","last_change":"2024-04-25T15:19:46.465000+0000","last_active":"2024-04-25T15:19:53.397172+0000","last_peered":"2024-04-25T15:19:53.397172+0000","last_clean":"2024-04-25T15:19:53.397172+0000","last_became_active":"2024-04-25T15:19:46.464680+0000","last_became_peered":"2024-04-25T15:19:46.464680+0000","last_unstale":"2024-04-25T15:19:53.397172+0000","last_undegraded":"2024-04-25T15:19:53.397172+0000","last_fullsized":"2024-04-25T15:19:53.397172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:35.597757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020764+0000","last_change":"2024-04-25T15:19:45.463067+0000","last_active":"2024-04-25T15:19:54.020764+0000","last_peered":"2024-04-25T15:19:54.020764+0000","last_clean":"2024-04-25T15:19:54.020764+0000","last_became_active":"2024-04-25T15:19:45.462611+0000","last_became_peered":"2024-04-25T15:19:45.462611+0000","last_unstale":"2024-04-25T15:19:54.020764+0000","last_undegraded":"2024-04-25T15:19:54.020764+0000","last_fullsized":"2024-04-25T15:19:54.020764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490099+0000","last_change":"2024-04-25T15:19:51.101470+0000","last_active":"2024-04-25T15:19:52.490099+0000","last_peered":"2024-04-25T15:19:52.490099+0000","last_clean":"2024-04-25T15:19:52.490099+0000","last_became_active":"2024-04-25T15:19:51.100991+0000","last_became_peered":"2024-04-25T15:19:51.100991+0000","last_unstale":"2024-04-25T15:19:52.490099+0000","last_undegraded":"2024-04-25T15:19:52.490099+0000","last_fullsized":"2024-04-25T15:19:52.490099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396886+0000","last_change":"2024-04-25T15:19:46.478228+0000","last_active":"2024-04-25T15:19:53.396886+0000","last_peered":"2024-04-25T15:19:53.396886+0000","last_clean":"2024-04-25T15:19:53.396886+0000","last_became_active":"2024-04-25T15:19:46.477939+0000","last_became_peered":"2024-04-25T15:19:46.477939+0000","last_unstale":"2024-04-25T15:19:53.396886+0000","last_undegraded":"2024-04-25T15:19:53.396886+0000","last_fullsized":"2024-04-25T15:19:53.396886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491269+0000","last_change":"2024-04-25T15:19:45.456572+0000","last_active":"2024-04-25T15:19:52.491269+0000","last_peered":"2024-04-25T15:19:52.491269+0000","last_clean":"2024-04-25T15:19:52.491269+0000","last_became_active":"2024-04-25T15:19:45.456238+0000","last_became_peered":"2024-04-25T15:19:45.456238+0000","last_unstale":"2024-04-25T15:19:52.491269+0000","last_undegraded":"2024-04-25T15:19:52.491269+0000","last_fullsized":"2024-04-25T15:19:52.491269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:09.326979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491720+0000","last_change":"2024-04-25T15:19:50.511879+0000","last_active":"2024-04-25T15:19:52.491720+0000","last_peered":"2024-04-25T15:19:52.491720+0000","last_clean":"2024-04-25T15:19:52.491720+0000","last_became_active":"2024-04-25T15:19:50.511380+0000","last_became_peered":"2024-04-25T15:19:50.511380+0000","last_unstale":"2024-04-25T15:19:52.491720+0000","last_undegraded":"2024-04-25T15:19:52.491720+0000","last_fullsized":"2024-04-25T15:19:52.491720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396446+0000","last_change":"2024-04-25T15:19:46.465026+0000","last_active":"2024-04-25T15:19:53.396446+0000","last_peered":"2024-04-25T15:19:53.396446+0000","last_clean":"2024-04-25T15:19:53.396446+0000","last_became_active":"2024-04-25T15:19:46.464703+0000","last_became_peered":"2024-04-25T15:19:46.464703+0000","last_unstale":"2024-04-25T15:19:53.396446+0000","last_undegraded":"2024-04-25T15:19:53.396446+0000","last_fullsized":"2024-04-25T15:19:53.396446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020165+0000","last_change":"2024-04-25T15:19:45.465017+0000","last_active":"2024-04-25T15:19:54.020165+0000","last_peered":"2024-04-25T15:19:54.020165+0000","last_clean":"2024-04-25T15:19:54.020165+0000","last_became_active":"2024-04-25T15:19:45.464691+0000","last_became_peered":"2024-04-25T15:19:45.464691+0000","last_unstale":"2024-04-25T15:19:54.020165+0000","last_undegraded":"2024-04-25T15:19:54.020165+0000","last_fullsized":"2024-04-25T15:19:54.020165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489222+0000","last_change":"2024-04-25T15:19:50.508942+0000","last_active":"2024-04-25T15:19:52.489222+0000","last_peered":"2024-04-25T15:19:52.489222+0000","last_clean":"2024-04-25T15:19:52.489222+0000","last_became_active":"2024-04-25T15:19:50.508482+0000","last_became_peered":"2024-04-25T15:19:50.508482+0000","last_unstale":"2024-04-25T15:19:52.489222+0000","last_undegraded":"2024-04-25T15:19:52.489222+0000","last_fullsized":"2024-04-25T15:19:52.489222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018243+0000","last_change":"2024-04-25T15:19:46.473708+0000","last_active":"2024-04-25T15:19:54.018243+0000","last_peered":"2024-04-25T15:19:54.018243+0000","last_clean":"2024-04-25T15:19:54.018243+0000","last_became_active":"2024-04-25T15:19:46.473372+0000","last_became_peered":"2024-04-25T15:19:46.473372+0000","last_unstale":"2024-04-25T15:19:54.018243+0000","last_undegraded":"2024-04-25T15:19:54.018243+0000","last_fullsized":"2024-04-25T15:19:54.018243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.469745+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.469458+0000","last_became_peered":"2024-04-25T15:19:45.469458+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020364+0000","last_change":"2024-04-25T15:19:51.101455+0000","last_active":"2024-04-25T15:19:54.020364+0000","last_peered":"2024-04-25T15:19:54.020364+0000","last_clean":"2024-04-25T15:19:54.020364+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:54.020364+0000","last_undegraded":"2024-04-25T15:19:54.020364+0000","last_fullsized":"2024-04-25T15:19:54.020364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:05.089655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396638+0000","last_change":"2024-04-25T15:19:46.465773+0000","last_active":"2024-04-25T15:19:53.396638+0000","last_peered":"2024-04-25T15:19:53.396638+0000","last_clean":"2024-04-25T15:19:53.396638+0000","last_became_active":"2024-04-25T15:19:46.465467+0000","last_became_peered":"2024-04-25T15:19:46.465467+0000","last_unstale":"2024-04-25T15:19:53.396638+0000","last_undegraded":"2024-04-25T15:19:53.396638+0000","last_fullsized":"2024-04-25T15:19:53.396638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491007+0000","last_change":"2024-04-25T15:19:45.463534+0000","last_active":"2024-04-25T15:19:52.491007+0000","last_peered":"2024-04-25T15:19:52.491007+0000","last_clean":"2024-04-25T15:19:52.491007+0000","last_became_active":"2024-04-25T15:19:45.463167+0000","last_became_peered":"2024-04-25T15:19:45.463167+0000","last_unstale":"2024-04-25T15:19:52.491007+0000","last_undegraded":"2024-04-25T15:19:52.491007+0000","last_fullsized":"2024-04-25T15:19:52.491007+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021379+0000","last_change":"2024-04-25T15:19:51.102717+0000","last_active":"2024-04-25T15:19:54.021379+0000","last_peered":"2024-04-25T15:19:54.021379+0000","last_clean":"2024-04-25T15:19:54.021379+0000","last_became_active":"2024-04-25T15:19:51.102247+0000","last_became_peered":"2024-04-25T15:19:51.102247+0000","last_unstale":"2024-04-25T15:19:54.021379+0000","last_undegraded":"2024-04-25T15:19:54.021379+0000","last_fullsized":"2024-04-25T15:19:54.021379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:14:00.372742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489038+0000","last_change":"2024-04-25T15:19:46.460776+0000","last_active":"2024-04-25T15:19:52.489038+0000","last_peered":"2024-04-25T15:19:52.489038+0000","last_clean":"2024-04-25T15:19:52.489038+0000","last_became_active":"2024-04-25T15:19:46.460453+0000","last_became_peered":"2024-04-25T15:19:46.460453+0000","last_unstale":"2024-04-25T15:19:52.489038+0000","last_undegraded":"2024-04-25T15:19:52.489038+0000","last_fullsized":"2024-04-25T15:19:52.489038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489102+0000","last_change":"2024-04-25T15:19:45.462749+0000","last_active":"2024-04-25T15:19:52.489102+0000","last_peered":"2024-04-25T15:19:52.489102+0000","last_clean":"2024-04-25T15:19:52.489102+0000","last_became_active":"2024-04-25T15:19:45.462417+0000","last_became_peered":"2024-04-25T15:19:45.462417+0000","last_unstale":"2024-04-25T15:19:52.489102+0000","last_undegraded":"2024-04-25T15:19:52.489102+0000","last_fullsized":"2024-04-25T15:19:52.489102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491884+0000","last_change":"2024-04-25T15:19:51.102630+0000","last_active":"2024-04-25T15:19:52.491884+0000","last_peered":"2024-04-25T15:19:52.491884+0000","last_clean":"2024-04-25T15:19:52.491884+0000","last_became_active":"2024-04-25T15:19:51.101912+0000","last_became_peered":"2024-04-25T15:19:51.101912+0000","last_unstale":"2024-04-25T15:19:52.491884+0000","last_undegraded":"2024-04-25T15:19:52.491884+0000","last_fullsized":"2024-04-25T15:19:52.491884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396712+0000","last_change":"2024-04-25T15:19:46.471583+0000","last_active":"2024-04-25T15:19:53.396712+0000","last_peered":"2024-04-25T15:19:53.396712+0000","last_clean":"2024-04-25T15:19:53.396712+0000","last_became_active":"2024-04-25T15:19:46.471286+0000","last_became_peered":"2024-04-25T15:19:46.471286+0000","last_unstale":"2024-04-25T15:19:53.396712+0000","last_undegraded":"2024-04-25T15:19:53.396712+0000","last_fullsized":"2024-04-25T15:19:53.396712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:55.948605+0000","last_change":"2024-04-25T15:19:45.464680+0000","last_active":"2024-04-25T15:19:55.948605+0000","last_peered":"2024-04-25T15:19:55.948605+0000","last_clean":"2024-04-25T15:19:55.948605+0000","last_became_active":"2024-04-25T15:19:45.463690+0000","last_became_peered":"2024-04-25T15:19:45.463690+0000","last_unstale":"2024-04-25T15:19:55.948605+0000","last_undegraded":"2024-04-25T15:19:55.948605+0000","last_fullsized":"2024-04-25T15:19:55.948605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809637+0000","last_change":"2024-04-25T15:19:50.509907+0000","last_active":"2024-04-25T15:19:53.809637+0000","last_peered":"2024-04-25T15:19:53.809637+0000","last_clean":"2024-04-25T15:19:53.809637+0000","last_became_active":"2024-04-25T15:19:50.508793+0000","last_became_peered":"2024-04-25T15:19:50.508793+0000","last_unstale":"2024-04-25T15:19:53.809637+0000","last_undegraded":"2024-04-25T15:19:53.809637+0000","last_fullsized":"2024-04-25T15:19:53.809637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490746+0000","last_change":"2024-04-25T15:19:46.471815+0000","last_active":"2024-04-25T15:19:52.490746+0000","last_peered":"2024-04-25T15:19:52.490746+0000","last_clean":"2024-04-25T15:19:52.490746+0000","last_became_active":"2024-04-25T15:19:46.471473+0000","last_became_peered":"2024-04-25T15:19:46.471473+0000","last_unstale":"2024-04-25T15:19:52.490746+0000","last_undegraded":"2024-04-25T15:19:52.490746+0000","last_fullsized":"2024-04-25T15:19:52.490746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809045+0000","last_change":"2024-04-25T15:19:45.472821+0000","last_active":"2024-04-25T15:19:53.809045+0000","last_peered":"2024-04-25T15:19:53.809045+0000","last_clean":"2024-04-25T15:19:53.809045+0000","last_became_active":"2024-04-25T15:19:45.472420+0000","last_became_peered":"2024-04-25T15:19:45.472420+0000","last_unstale":"2024-04-25T15:19:53.809045+0000","last_undegraded":"2024-04-25T15:19:53.809045+0000","last_fullsized":"2024-04-25T15:19:53.809045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492136+0000","last_change":"2024-04-25T15:19:50.509990+0000","last_active":"2024-04-25T15:19:52.492136+0000","last_peered":"2024-04-25T15:19:52.492136+0000","last_clean":"2024-04-25T15:19:52.492136+0000","last_became_active":"2024-04-25T15:19:50.509340+0000","last_became_peered":"2024-04-25T15:19:50.509340+0000","last_unstale":"2024-04-25T15:19:52.492136+0000","last_undegraded":"2024-04-25T15:19:52.492136+0000","last_fullsized":"2024-04-25T15:19:52.492136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489275+0000","last_change":"2024-04-25T15:19:46.485819+0000","last_active":"2024-04-25T15:19:52.489275+0000","last_peered":"2024-04-25T15:19:52.489275+0000","last_clean":"2024-04-25T15:19:52.489275+0000","last_became_active":"2024-04-25T15:19:46.485430+0000","last_became_peered":"2024-04-25T15:19:46.485430+0000","last_unstale":"2024-04-25T15:19:52.489275+0000","last_undegraded":"2024-04-25T15:19:52.489275+0000","last_fullsized":"2024-04-25T15:19:52.489275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490843+0000","last_change":"2024-04-25T15:19:45.461658+0000","last_active":"2024-04-25T15:19:52.490843+0000","last_peered":"2024-04-25T15:19:52.490843+0000","last_clean":"2024-04-25T15:19:52.490843+0000","last_became_active":"2024-04-25T15:19:45.461346+0000","last_became_peered":"2024-04-25T15:19:45.461346+0000","last_unstale":"2024-04-25T15:19:52.490843+0000","last_undegraded":"2024-04-25T15:19:52.490843+0000","last_fullsized":"2024-04-25T15:19:52.490843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021048+0000","last_change":"2024-04-25T15:19:50.518359+0000","last_active":"2024-04-25T15:19:54.021048+0000","last_peered":"2024-04-25T15:19:54.021048+0000","last_clean":"2024-04-25T15:19:54.021048+0000","last_became_active":"2024-04-25T15:19:50.517856+0000","last_became_peered":"2024-04-25T15:19:50.517856+0000","last_unstale":"2024-04-25T15:19:54.021048+0000","last_undegraded":"2024-04-25T15:19:54.021048+0000","last_fullsized":"2024-04-25T15:19:54.021048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:08:18.271358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490731+0000","last_change":"2024-04-25T15:19:46.469634+0000","last_active":"2024-04-25T15:19:52.490731+0000","last_peered":"2024-04-25T15:19:52.490731+0000","last_clean":"2024-04-25T15:19:52.490731+0000","last_became_active":"2024-04-25T15:19:46.468979+0000","last_became_peered":"2024-04-25T15:19:46.468979+0000","last_unstale":"2024-04-25T15:19:52.490731+0000","last_undegraded":"2024-04-25T15:19:52.490731+0000","last_fullsized":"2024-04-25T15:19:52.490731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017779+0000","last_change":"2024-04-25T15:19:45.471612+0000","last_active":"2024-04-25T15:19:54.017779+0000","last_peered":"2024-04-25T15:19:54.017779+0000","last_clean":"2024-04-25T15:19:54.017779+0000","last_became_active":"2024-04-25T15:19:45.471289+0000","last_became_peered":"2024-04-25T15:19:45.471289+0000","last_unstale":"2024-04-25T15:19:54.017779+0000","last_undegraded":"2024-04-25T15:19:54.017779+0000","last_fullsized":"2024-04-25T15:19:54.017779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020544+0000","last_change":"2024-04-25T15:19:51.105778+0000","last_active":"2024-04-25T15:19:54.020544+0000","last_peered":"2024-04-25T15:19:54.020544+0000","last_clean":"2024-04-25T15:19:54.020544+0000","last_became_active":"2024-04-25T15:19:51.105435+0000","last_became_peered":"2024-04-25T15:19:51.105435+0000","last_unstale":"2024-04-25T15:19:54.020544+0000","last_undegraded":"2024-04-25T15:19:54.020544+0000","last_fullsized":"2024-04-25T15:19:54.020544+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:09:15.216797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489181+0000","last_change":"2024-04-25T15:19:51.101540+0000","last_active":"2024-04-25T15:19:52.489181+0000","last_peered":"2024-04-25T15:19:52.489181+0000","last_clean":"2024-04-25T15:19:52.489181+0000","last_became_active":"2024-04-25T15:19:51.101126+0000","last_became_peered":"2024-04-25T15:19:51.101126+0000","last_unstale":"2024-04-25T15:19:52.489181+0000","last_undegraded":"2024-04-25T15:19:52.489181+0000","last_fullsized":"2024-04-25T15:19:52.489181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809353+0000","last_change":"2024-04-25T15:19:46.470144+0000","last_active":"2024-04-25T15:19:53.809353+0000","last_peered":"2024-04-25T15:19:53.809353+0000","last_clean":"2024-04-25T15:19:53.809353+0000","last_became_active":"2024-04-25T15:19:46.469786+0000","last_became_peered":"2024-04-25T15:19:46.469786+0000","last_unstale":"2024-04-25T15:19:53.809353+0000","last_undegraded":"2024-04-25T15:19:53.809353+0000","last_fullsized":"2024-04-25T15:19:53.809353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490304+0000","last_change":"2024-04-25T15:19:45.466078+0000","last_active":"2024-04-25T15:19:52.490304+0000","last_peered":"2024-04-25T15:19:52.490304+0000","last_clean":"2024-04-25T15:19:52.490304+0000","last_became_active":"2024-04-25T15:19:45.465744+0000","last_became_peered":"2024-04-25T15:19:45.465744+0000","last_unstale":"2024-04-25T15:19:52.490304+0000","last_undegraded":"2024-04-25T15:19:52.490304+0000","last_fullsized":"2024-04-25T15:19:52.490304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018744+0000","last_change":"2024-04-25T15:19:50.517045+0000","last_active":"2024-04-25T15:19:54.018744+0000","last_peered":"2024-04-25T15:19:54.018744+0000","last_clean":"2024-04-25T15:19:54.018744+0000","last_became_active":"2024-04-25T15:19:50.516571+0000","last_became_peered":"2024-04-25T15:19:50.516571+0000","last_unstale":"2024-04-25T15:19:54.018744+0000","last_undegraded":"2024-04-25T15:19:54.018744+0000","last_fullsized":"2024-04-25T15:19:54.018744+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020866+0000","last_change":"2024-04-25T15:19:46.472281+0000","last_active":"2024-04-25T15:19:54.020866+0000","last_peered":"2024-04-25T15:19:54.020866+0000","last_clean":"2024-04-25T15:19:54.020866+0000","last_became_active":"2024-04-25T15:19:46.471957+0000","last_became_peered":"2024-04-25T15:19:46.471957+0000","last_unstale":"2024-04-25T15:19:54.020866+0000","last_undegraded":"2024-04-25T15:19:54.020866+0000","last_fullsized":"2024-04-25T15:19:54.020866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021072+0000","last_change":"2024-04-25T15:19:45.461678+0000","last_active":"2024-04-25T15:19:54.021072+0000","last_peered":"2024-04-25T15:19:54.021072+0000","last_clean":"2024-04-25T15:19:54.021072+0000","last_became_active":"2024-04-25T15:19:45.461365+0000","last_became_peered":"2024-04-25T15:19:45.461365+0000","last_unstale":"2024-04-25T15:19:54.021072+0000","last_undegraded":"2024-04-25T15:19:54.021072+0000","last_fullsized":"2024-04-25T15:19:54.021072+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.468983+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.468673+0000","last_became_peered":"2024-04-25T15:19:45.468673+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489044+0000","last_change":"2024-04-25T15:19:46.459915+0000","last_active":"2024-04-25T15:19:52.489044+0000","last_peered":"2024-04-25T15:19:52.489044+0000","last_clean":"2024-04-25T15:19:52.489044+0000","last_became_active":"2024-04-25T15:19:46.459596+0000","last_became_peered":"2024-04-25T15:19:46.459596+0000","last_unstale":"2024-04-25T15:19:52.489044+0000","last_undegraded":"2024-04-25T15:19:52.489044+0000","last_fullsized":"2024-04-25T15:19:52.489044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020992+0000","last_change":"2024-04-25T15:19:50.517033+0000","last_active":"2024-04-25T15:19:54.020992+0000","last_peered":"2024-04-25T15:19:54.020992+0000","last_clean":"2024-04-25T15:19:54.020992+0000","last_became_active":"2024-04-25T15:19:50.516587+0000","last_became_peered":"2024-04-25T15:19:50.516587+0000","last_unstale":"2024-04-25T15:19:54.020992+0000","last_undegraded":"2024-04-25T15:19:54.020992+0000","last_fullsized":"2024-04-25T15:19:54.020992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:50.341053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019861+0000","last_change":"2024-04-25T15:19:45.466373+0000","last_active":"2024-04-25T15:19:54.019861+0000","last_peered":"2024-04-25T15:19:54.019861+0000","last_clean":"2024-04-25T15:19:54.019861+0000","last_became_active":"2024-04-25T15:19:45.466054+0000","last_became_peered":"2024-04-25T15:19:45.466054+0000","last_unstale":"2024-04-25T15:19:54.019861+0000","last_undegraded":"2024-04-25T15:19:54.019861+0000","last_fullsized":"2024-04-25T15:19:54.019861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:33.631245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491098+0000","last_change":"2024-04-25T15:19:46.471011+0000","last_active":"2024-04-25T15:19:52.491098+0000","last_peered":"2024-04-25T15:19:52.491098+0000","last_clean":"2024-04-25T15:19:52.491098+0000","last_became_active":"2024-04-25T15:19:46.470620+0000","last_became_peered":"2024-04-25T15:19:46.470620+0000","last_unstale":"2024-04-25T15:19:52.491098+0000","last_undegraded":"2024-04-25T15:19:52.491098+0000","last_fullsized":"2024-04-25T15:19:52.491098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:15.434932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021206+0000","last_change":"2024-04-25T15:19:50.517337+0000","last_active":"2024-04-25T15:19:54.021206+0000","last_peered":"2024-04-25T15:19:54.021206+0000","last_clean":"2024-04-25T15:19:54.021206+0000","last_became_active":"2024-04-25T15:19:50.516848+0000","last_became_peered":"2024-04-25T15:19:50.516848+0000","last_unstale":"2024-04-25T15:19:54.021206+0000","last_undegraded":"2024-04-25T15:19:54.021206+0000","last_fullsized":"2024-04-25T15:19:54.021206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:21:22.692161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018022+0000","last_change":"2024-04-25T15:19:46.476012+0000","last_active":"2024-04-25T15:19:54.018022+0000","last_peered":"2024-04-25T15:19:54.018022+0000","last_clean":"2024-04-25T15:19:54.018022+0000","last_became_active":"2024-04-25T15:19:46.475700+0000","last_became_peered":"2024-04-25T15:19:46.475700+0000","last_unstale":"2024-04-25T15:19:54.018022+0000","last_undegraded":"2024-04-25T15:19:54.018022+0000","last_fullsized":"2024-04-25T15:19:54.018022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020188+0000","last_change":"2024-04-25T15:19:46.484292+0000","last_active":"2024-04-25T15:19:54.020188+0000","last_peered":"2024-04-25T15:19:54.020188+0000","last_clean":"2024-04-25T15:19:54.020188+0000","last_became_active":"2024-04-25T15:19:46.483963+0000","last_became_peered":"2024-04-25T15:19:46.483963+0000","last_unstale":"2024-04-25T15:19:54.020188+0000","last_undegraded":"2024-04-25T15:19:54.020188+0000","last_fullsized":"2024-04-25T15:19:54.020188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490912+0000","last_change":"2024-04-25T15:19:46.484902+0000","last_active":"2024-04-25T15:19:52.490912+0000","last_peered":"2024-04-25T15:19:52.490912+0000","last_clean":"2024-04-25T15:19:52.490912+0000","last_became_active":"2024-04-25T15:19:46.482984+0000","last_became_peered":"2024-04-25T15:19:46.482984+0000","last_unstale":"2024-04-25T15:19:52.490912+0000","last_undegraded":"2024-04-25T15:19:52.490912+0000","last_fullsized":"2024-04-25T15:19:52.490912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488966+0000","last_change":"2024-04-25T15:19:46.468583+0000","last_active":"2024-04-25T15:19:52.488966+0000","last_peered":"2024-04-25T15:19:52.488966+0000","last_clean":"2024-04-25T15:19:52.488966+0000","last_became_active":"2024-04-25T15:19:46.468260+0000","last_became_peered":"2024-04-25T15:19:46.468260+0000","last_unstale":"2024-04-25T15:19:52.488966+0000","last_undegraded":"2024-04-25T15:19:52.488966+0000","last_fullsized":"2024-04-25T15:19:52.488966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021455+0000","last_change":"2024-04-25T15:19:46.478008+0000","last_active":"2024-04-25T15:19:54.021455+0000","last_peered":"2024-04-25T15:19:54.021455+0000","last_clean":"2024-04-25T15:19:54.021455+0000","last_became_active":"2024-04-25T15:19:46.477687+0000","last_became_peered":"2024-04-25T15:19:46.477687+0000","last_unstale":"2024-04-25T15:19:54.021455+0000","last_undegraded":"2024-04-25T15:19:54.021455+0000","last_fullsized":"2024-04-25T15:19:54.021455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809819+0000","last_change":"2024-04-25T15:19:46.475994+0000","last_active":"2024-04-25T15:19:53.809819+0000","last_peered":"2024-04-25T15:19:53.809819+0000","last_clean":"2024-04-25T15:19:53.809819+0000","last_became_active":"2024-04-25T15:19:46.475566+0000","last_became_peered":"2024-04-25T15:19:46.475566+0000","last_unstale":"2024-04-25T15:19:53.809819+0000","last_undegraded":"2024-04-25T15:19:53.809819+0000","last_fullsized":"2024-04-25T15:19:53.809819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018933+0000","last_change":"2024-04-25T15:19:46.480931+0000","last_active":"2024-04-25T15:19:54.018933+0000","last_peered":"2024-04-25T15:19:54.018933+0000","last_clean":"2024-04-25T15:19:54.018933+0000","last_became_active":"2024-04-25T15:19:46.480607+0000","last_became_peered":"2024-04-25T15:19:46.480607+0000","last_unstale":"2024-04-25T15:19:54.018933+0000","last_undegraded":"2024-04-25T15:19:54.018933+0000","last_fullsized":"2024-04-25T15:19:54.018933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020473+0000","last_change":"2024-04-25T15:19:46.478746+0000","last_active":"2024-04-25T15:19:54.020473+0000","last_peered":"2024-04-25T15:19:54.020473+0000","last_clean":"2024-04-25T15:19:54.020473+0000","last_became_active":"2024-04-25T15:19:46.478389+0000","last_became_peered":"2024-04-25T15:19:46.478389+0000","last_unstale":"2024-04-25T15:19:54.020473+0000","last_undegraded":"2024-04-25T15:19:54.020473+0000","last_fullsized":"2024-04-25T15:19:54.020473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490185+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.490185+0000","last_peered":"2024-04-25T15:19:52.490185+0000","last_clean":"2024-04-25T15:19:52.490185+0000","last_became_active":"2024-04-25T15:19:46.482115+0000","last_became_peered":"2024-04-25T15:19:46.482115+0000","last_unstale":"2024-04-25T15:19:52.490185+0000","last_undegraded":"2024-04-25T15:19:52.490185+0000","last_fullsized":"2024-04-25T15:19:52.490185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020755+0000","last_change":"2024-04-25T15:19:46.477554+0000","last_active":"2024-04-25T15:19:54.020755+0000","last_peered":"2024-04-25T15:19:54.020755+0000","last_clean":"2024-04-25T15:19:54.020755+0000","last_became_active":"2024-04-25T15:19:46.477240+0000","last_became_peered":"2024-04-25T15:19:46.477240+0000","last_unstale":"2024-04-25T15:19:54.020755+0000","last_undegraded":"2024-04-25T15:19:54.020755+0000","last_fullsized":"2024-04-25T15:19:54.020755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490811+0000","last_change":"2024-04-25T15:19:46.486093+0000","last_active":"2024-04-25T15:19:52.490811+0000","last_peered":"2024-04-25T15:19:52.490811+0000","last_clean":"2024-04-25T15:19:52.490811+0000","last_became_active":"2024-04-25T15:19:46.484171+0000","last_became_peered":"2024-04-25T15:19:46.484171+0000","last_unstale":"2024-04-25T15:19:52.490811+0000","last_undegraded":"2024-04-25T15:19:52.490811+0000","last_fullsized":"2024-04-25T15:19:52.490811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488875+0000","last_change":"2024-04-25T15:19:46.472207+0000","last_active":"2024-04-25T15:19:52.488875+0000","last_peered":"2024-04-25T15:19:52.488875+0000","last_clean":"2024-04-25T15:19:52.488875+0000","last_became_active":"2024-04-25T15:19:46.471780+0000","last_became_peered":"2024-04-25T15:19:46.471780+0000","last_unstale":"2024-04-25T15:19:52.488875+0000","last_undegraded":"2024-04-25T15:19:52.488875+0000","last_fullsized":"2024-04-25T15:19:52.488875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018502+0000","last_change":"2024-04-25T15:19:46.477605+0000","last_active":"2024-04-25T15:19:54.018502+0000","last_peered":"2024-04-25T15:19:54.018502+0000","last_clean":"2024-04-25T15:19:54.018502+0000","last_became_active":"2024-04-25T15:19:46.477324+0000","last_became_peered":"2024-04-25T15:19:46.477324+0000","last_unstale":"2024-04-25T15:19:54.018502+0000","last_undegraded":"2024-04-25T15:19:54.018502+0000","last_fullsized":"2024-04-25T15:19:54.018502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397007+0000","last_change":"2024-04-25T15:19:46.468508+0000","last_active":"2024-04-25T15:19:53.397007+0000","last_peered":"2024-04-25T15:19:53.397007+0000","last_clean":"2024-04-25T15:19:53.397007+0000","last_became_active":"2024-04-25T15:19:46.468170+0000","last_became_peered":"2024-04-25T15:19:46.468170+0000","last_unstale":"2024-04-25T15:19:53.397007+0000","last_undegraded":"2024-04-25T15:19:53.397007+0000","last_fullsized":"2024-04-25T15:19:53.397007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:00.962714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020658+0000","last_change":"2024-04-25T15:19:46.475011+0000","last_active":"2024-04-25T15:19:54.020658+0000","last_peered":"2024-04-25T15:19:54.020658+0000","last_clean":"2024-04-25T15:19:54.020658+0000","last_became_active":"2024-04-25T15:19:46.474637+0000","last_became_peered":"2024-04-25T15:19:46.474637+0000","last_unstale":"2024-04-25T15:19:54.020658+0000","last_undegraded":"2024-04-25T15:19:54.020658+0000","last_fullsized":"2024-04-25T15:19:54.020658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491965+0000","last_change":"2024-04-25T15:19:46.483654+0000","last_active":"2024-04-25T15:19:52.491965+0000","last_peered":"2024-04-25T15:19:52.491965+0000","last_clean":"2024-04-25T15:19:52.491965+0000","last_became_active":"2024-04-25T15:19:46.483291+0000","last_became_peered":"2024-04-25T15:19:46.483291+0000","last_unstale":"2024-04-25T15:19:52.491965+0000","last_undegraded":"2024-04-25T15:19:52.491965+0000","last_fullsized":"2024-04-25T15:19:52.491965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490226+0000","last_change":"2024-04-25T15:19:46.483513+0000","last_active":"2024-04-25T15:19:52.490226+0000","last_peered":"2024-04-25T15:19:52.490226+0000","last_clean":"2024-04-25T15:19:52.490226+0000","last_became_active":"2024-04-25T15:19:46.483158+0000","last_became_peered":"2024-04-25T15:19:46.483158+0000","last_unstale":"2024-04-25T15:19:52.490226+0000","last_undegraded":"2024-04-25T15:19:52.490226+0000","last_fullsized":"2024-04-25T15:19:52.490226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809438+0000","last_change":"2024-04-25T15:19:46.479148+0000","last_active":"2024-04-25T15:19:53.809438+0000","last_peered":"2024-04-25T15:19:53.809438+0000","last_clean":"2024-04-25T15:19:53.809438+0000","last_became_active":"2024-04-25T15:19:46.478829+0000","last_became_peered":"2024-04-25T15:19:46.478829+0000","last_unstale":"2024-04-25T15:19:53.809438+0000","last_undegraded":"2024-04-25T15:19:53.809438+0000","last_fullsized":"2024-04-25T15:19:53.809438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489010+0000","last_change":"2024-04-25T15:19:46.471858+0000","last_active":"2024-04-25T15:19:52.489010+0000","last_peered":"2024-04-25T15:19:52.489010+0000","last_clean":"2024-04-25T15:19:52.489010+0000","last_became_active":"2024-04-25T15:19:46.471517+0000","last_became_peered":"2024-04-25T15:19:46.471517+0000","last_unstale":"2024-04-25T15:19:52.489010+0000","last_undegraded":"2024-04-25T15:19:52.489010+0000","last_fullsized":"2024-04-25T15:19:52.489010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:58.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020190+0000","last_change":"2024-04-25T15:19:46.473809+0000","last_active":"2024-04-25T15:19:54.020190+0000","last_peered":"2024-04-25T15:19:54.020190+0000","last_clean":"2024-04-25T15:19:54.020190+0000","last_became_active":"2024-04-25T15:19:46.473479+0000","last_became_peered":"2024-04-25T15:19:46.473479+0000","last_unstale":"2024-04-25T15:19:54.020190+0000","last_undegraded":"2024-04-25T15:19:54.020190+0000","last_fullsized":"2024-04-25T15:19:54.020190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490638+0000","last_change":"2024-04-25T15:19:46.473643+0000","last_active":"2024-04-25T15:19:52.490638+0000","last_peered":"2024-04-25T15:19:52.490638+0000","last_clean":"2024-04-25T15:19:52.490638+0000","last_became_active":"2024-04-25T15:19:46.473190+0000","last_became_peered":"2024-04-25T15:19:46.473190+0000","last_unstale":"2024-04-25T15:19:52.490638+0000","last_undegraded":"2024-04-25T15:19:52.490638+0000","last_fullsized":"2024-04-25T15:19:52.490638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018544+0000","last_change":"2024-04-25T15:19:46.488270+0000","last_active":"2024-04-25T15:19:54.018544+0000","last_peered":"2024-04-25T15:19:54.018544+0000","last_clean":"2024-04-25T15:19:54.018544+0000","last_became_active":"2024-04-25T15:19:46.487935+0000","last_became_peered":"2024-04-25T15:19:46.487935+0000","last_unstale":"2024-04-25T15:19:54.018544+0000","last_undegraded":"2024-04-25T15:19:54.018544+0000","last_fullsized":"2024-04-25T15:19:54.018544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018847+0000","last_change":"2024-04-25T15:19:46.488350+0000","last_active":"2024-04-25T15:19:54.018847+0000","last_peered":"2024-04-25T15:19:54.018847+0000","last_clean":"2024-04-25T15:19:54.018847+0000","last_became_active":"2024-04-25T15:19:46.487997+0000","last_became_peered":"2024-04-25T15:19:46.487997+0000","last_unstale":"2024-04-25T15:19:54.018847+0000","last_undegraded":"2024-04-25T15:19:54.018847+0000","last_fullsized":"2024-04-25T15:19:54.018847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397147+0000","last_change":"2024-04-25T15:19:46.486142+0000","last_active":"2024-04-25T15:19:53.397147+0000","last_peered":"2024-04-25T15:19:53.397147+0000","last_clean":"2024-04-25T15:19:53.397147+0000","last_became_active":"2024-04-25T15:19:46.485774+0000","last_became_peered":"2024-04-25T15:19:46.485774+0000","last_unstale":"2024-04-25T15:19:53.397147+0000","last_undegraded":"2024-04-25T15:19:53.397147+0000","last_fullsized":"2024-04-25T15:19:53.397147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:16.297767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020777+0000","last_change":"2024-04-25T15:19:46.486137+0000","last_active":"2024-04-25T15:19:54.020777+0000","last_peered":"2024-04-25T15:19:54.020777+0000","last_clean":"2024-04-25T15:19:54.020777+0000","last_became_active":"2024-04-25T15:19:46.485814+0000","last_became_peered":"2024-04-25T15:19:46.485814+0000","last_unstale":"2024-04-25T15:19:54.020777+0000","last_undegraded":"2024-04-25T15:19:54.020777+0000","last_fullsized":"2024-04-25T15:19:54.020777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.469644+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469320+0000","last_became_peered":"2024-04-25T15:19:46.469320+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490739+0000","last_change":"2024-04-25T15:19:46.486357+0000","last_active":"2024-04-25T15:19:52.490739+0000","last_peered":"2024-04-25T15:19:52.490739+0000","last_clean":"2024-04-25T15:19:52.490739+0000","last_became_active":"2024-04-25T15:19:46.486016+0000","last_became_peered":"2024-04-25T15:19:46.486016+0000","last_unstale":"2024-04-25T15:19:52.490739+0000","last_undegraded":"2024-04-25T15:19:52.490739+0000","last_fullsized":"2024-04-25T15:19:52.490739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.470290+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469965+0000","last_became_peered":"2024-04-25T15:19:46.469965+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491250+0000","last_change":"2024-04-25T15:19:46.483532+0000","last_active":"2024-04-25T15:19:52.491250+0000","last_peered":"2024-04-25T15:19:52.491250+0000","last_clean":"2024-04-25T15:19:52.491250+0000","last_became_active":"2024-04-25T15:19:46.483242+0000","last_became_peered":"2024-04-25T15:19:46.483242+0000","last_unstale":"2024-04-25T15:19:52.491250+0000","last_undegraded":"2024-04-25T15:19:52.491250+0000","last_fullsized":"2024-04-25T15:19:52.491250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489031+0000","last_change":"2024-04-25T15:19:46.467386+0000","last_active":"2024-04-25T15:19:52.489031+0000","last_peered":"2024-04-25T15:19:52.489031+0000","last_clean":"2024-04-25T15:19:52.489031+0000","last_became_active":"2024-04-25T15:19:46.467034+0000","last_became_peered":"2024-04-25T15:19:46.467034+0000","last_unstale":"2024-04-25T15:19:52.489031+0000","last_undegraded":"2024-04-25T15:19:52.489031+0000","last_fullsized":"2024-04-25T15:19:52.489031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809963+0000","last_change":"2024-04-25T15:19:46.472318+0000","last_active":"2024-04-25T15:19:53.809963+0000","last_peered":"2024-04-25T15:19:53.809963+0000","last_clean":"2024-04-25T15:19:53.809963+0000","last_became_active":"2024-04-25T15:19:46.471982+0000","last_became_peered":"2024-04-25T15:19:46.471982+0000","last_unstale":"2024-04-25T15:19:53.809963+0000","last_undegraded":"2024-04-25T15:19:53.809963+0000","last_fullsized":"2024-04-25T15:19:53.809963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397045+0000","last_change":"2024-04-25T15:19:46.473366+0000","last_active":"2024-04-25T15:19:53.397045+0000","last_peered":"2024-04-25T15:19:53.397045+0000","last_clean":"2024-04-25T15:19:53.397045+0000","last_became_active":"2024-04-25T15:19:46.472969+0000","last_became_peered":"2024-04-25T15:19:46.472969+0000","last_unstale":"2024-04-25T15:19:53.397045+0000","last_undegraded":"2024-04-25T15:19:53.397045+0000","last_fullsized":"2024-04-25T15:19:53.397045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:20:14.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:20:14.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.256+0000 7f40324e0640 1 Processor -- start 2024-04-25T15:20:14.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.256+0000 7f40324e0640 1 -- start start 2024-04-25T15:20:14.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f40324e0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f402c06ee60 0x7f402c0b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f40324e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f40324e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f402c164800 0x7f402c160830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f40324e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f402c08d730 con 0x7f402c164800 2024-04-25T15:20:14.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f40324e0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f402c06f440 con 0x7f402c06ee60 2024-04-25T15:20:14.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f40324e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f402c0b8eb0 con 0x7f402c163eb0 2024-04-25T15:20:14.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f402c164800 0x7f402c160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f402c164800 0x7f402c160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60320/0 (socket says 172.21.15.146:60320) 2024-04-25T15:20:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 -- 172.21.15.146:0/4094223173 learned_addr learned my addr 172.21.15.146:0/4094223173 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f402b7fe640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f402bfff640 1 --2- 172.21.15.146:0/4094223173 >> v2:172.21.15.177:3300/0 conn(0x7f402c06ee60 0x7f402c0b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 -- 172.21.15.146:0/4094223173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 msgr2=0x7f402c1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 -- 172.21.15.146:0/4094223173 >> v2:172.21.15.177:3300/0 conn(0x7f402c06ee60 msgr2=0x7f402c0b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 --2- 172.21.15.146:0/4094223173 >> v2:172.21.15.177:3300/0 conn(0x7f402c06ee60 0x7f402c0b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f4030a7d640 1 -- 172.21.15.146:0/4094223173 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c08ce00 con 0x7f402c164800 2024-04-25T15:20:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.258+0000 7f402b7fe640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.259+0000 7f4030a7d640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c164800 0x7f402c160830 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f402c08c9d0 tx=0x7f402402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109e71e462152c40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.259+0000 7f402affd640 1 -- 172.21.15.146:0/4094223173 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024036070 con 0x7f402c164800 2024-04-25T15:20:14.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.259+0000 7f402affd640 1 -- 172.21.15.146:0/4094223173 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4024031600 con 0x7f402c164800 2024-04-25T15:20:14.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.259+0000 7f402affd640 1 -- 172.21.15.146:0/4094223173 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40240319d0 con 0x7f402c164800 2024-04-25T15:20:14.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 -- 172.21.15.146:0/4094223173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c164800 msgr2=0x7f402c160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c164800 0x7f402c160830 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f402c08c9d0 tx=0x7f402402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 -- 172.21.15.146:0/4094223173 shutdown_connections 2024-04-25T15:20:14.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c164800 0x7f402c160830 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 --2- 172.21.15.146:0/4094223173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 --2- 172.21.15.146:0/4094223173 >> v2:172.21.15.177:3300/0 conn(0x7f402c06ee60 0x7f402c0b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 -- 172.21.15.146:0/4094223173 >> 172.21.15.146:0/4094223173 conn(0x7f402c0b52e0 msgr2=0x7f402c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:14.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 -- 172.21.15.146:0/4094223173 shutdown_connections 2024-04-25T15:20:14.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.260+0000 7f40324e0640 1 -- 172.21.15.146:0/4094223173 wait complete. 2024-04-25T15:20:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.262+0000 7f40324e0640 1 Processor -- start 2024-04-25T15:20:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.262+0000 7f40324e0640 1 -- start start 2024-04-25T15:20:14.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f40324e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 0x7f402c1eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f40324e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f40324e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f402c164800 0x7f402c0b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f40324e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f402c08dbc0 con 0x7f402c06ee60 2024-04-25T15:20:14.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f40324e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f402c08d3c0 con 0x7f402c164800 2024-04-25T15:20:14.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f40324e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f402c164c10 con 0x7f402c163eb0 2024-04-25T15:20:14.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f402bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 0x7f402c1eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f402bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 0x7f402c1eb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60324/0 (socket says 172.21.15.146:60324) 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f402bfff640 1 -- 172.21.15.146:0/1038950470 learned_addr learned my addr 172.21.15.146:0/1038950470 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f402b7fe640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.263+0000 7f4030a7d640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f402c164800 0x7f402c0b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f402bfff640 1 -- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 msgr2=0x7f402c150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f402bfff640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f402bfff640 1 -- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f402c164800 msgr2=0x7f402c0b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f402bfff640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f402c164800 0x7f402c0b8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f402bfff640 1 -- 172.21.15.146:0/1038950470 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c0b78b0 con 0x7f402c06ee60 2024-04-25T15:20:14.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f402bfff640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 0x7f402c1eb0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f401c00d830 tx=0x7f401c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c0096a0 con 0x7f402c06ee60 2024-04-25T15:20:14.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f401c019070 con 0x7f402c06ee60 2024-04-25T15:20:14.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402c0b7b90 con 0x7f402c06ee60 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c009a60 con 0x7f402c06ee60 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.264+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402c1eabc0 con 0x7f402c06ee60 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.265+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f402c0d2b10 con 0x7f402c06ee60 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.266+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f401c01e460 con 0x7f402c06ee60 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.266+0000 7f4028ff9640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f40180744e0 0x7f40180769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.266+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f401c095a00 con 0x7f402c06ee60 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.266+0000 7f402b7fe640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f40180744e0 0x7f40180769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.266+0000 7f402b7fe640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f40180744e0 0x7f40180769f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f402c071790 tx=0x7f4020008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.271+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f402c0d2b10 con 0x7f402c06ee60 2024-04-25T15:20:14.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.440+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f402c06a0e0 con 0x7f40180744e0 2024-04-25T15:20:14.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.448+0000 7f4028ff9640 1 -- 172.21.15.146:0/1038950470 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332522 (secure 0 0 0) 0x7f402c06a0e0 con 0x7f40180744e0 2024-04-25T15:20:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:14.455 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f40180744e0 msgr2=0x7f40180769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f40180744e0 0x7f40180769f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f402c071790 tx=0x7f4020008040 comp rx=0 tx=0).stop 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 msgr2=0x7f402c1eb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 0x7f402c1eb0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f401c00d830 tx=0x7f401c00dcf0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 shutdown_connections 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f40180744e0 0x7f40180769f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f402c164800 0x7f402c0b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f402c163eb0 0x7f402c150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 --2- 172.21.15.146:0/1038950470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c06ee60 0x7f402c1eb0e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 >> 172.21.15.146:0/1038950470 conn(0x7f402c0b52e0 msgr2=0x7f402c072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:14.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 shutdown_connections 2024-04-25T15:20:14.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.452+0000 7f40324e0640 1 -- 172.21.15.146:0/1038950470 wait complete. 2024-04-25T15:20:14.465 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-25T15:20:13.993556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003220"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489335+0000","last_change":"2024-04-25T15:19:45.467889+0000","last_active":"2024-04-25T15:19:52.489335+0000","last_peered":"2024-04-25T15:19:52.489335+0000","last_clean":"2024-04-25T15:19:52.489335+0000","last_became_active":"2024-04-25T15:19:45.467565+0000","last_became_peered":"2024-04-25T15:19:45.467565+0000","last_unstale":"2024-04-25T15:19:52.489335+0000","last_undegraded":"2024-04-25T15:19:52.489335+0000","last_fullsized":"2024-04-25T15:19:52.489335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020856+0000","last_change":"2024-04-25T15:19:46.476186+0000","last_active":"2024-04-25T15:19:54.020856+0000","last_peered":"2024-04-25T15:19:54.020856+0000","last_clean":"2024-04-25T15:19:54.020856+0000","last_became_active":"2024-04-25T15:19:46.475865+0000","last_became_peered":"2024-04-25T15:19:46.475865+0000","last_unstale":"2024-04-25T15:19:54.020856+0000","last_undegraded":"2024-04-25T15:19:54.020856+0000","last_fullsized":"2024-04-25T15:19:54.020856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020377+0000","last_change":"2024-04-25T15:19:51.102406+0000","last_active":"2024-04-25T15:19:54.020377+0000","last_peered":"2024-04-25T15:19:54.020377+0000","last_clean":"2024-04-25T15:19:54.020377+0000","last_became_active":"2024-04-25T15:19:51.101849+0000","last_became_peered":"2024-04-25T15:19:51.101849+0000","last_unstale":"2024-04-25T15:19:54.020377+0000","last_undegraded":"2024-04-25T15:19:54.020377+0000","last_fullsized":"2024-04-25T15:19:54.020377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:58.329289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020474+0000","last_change":"2024-04-25T15:19:45.461077+0000","last_active":"2024-04-25T15:19:54.020474+0000","last_peered":"2024-04-25T15:19:54.020474+0000","last_clean":"2024-04-25T15:19:54.020474+0000","last_became_active":"2024-04-25T15:19:45.460662+0000","last_became_peered":"2024-04-25T15:19:45.460662+0000","last_unstale":"2024-04-25T15:19:54.020474+0000","last_undegraded":"2024-04-25T15:19:54.020474+0000","last_fullsized":"2024-04-25T15:19:54.020474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489205+0000","last_change":"2024-04-25T15:19:46.486683+0000","last_active":"2024-04-25T15:19:52.489205+0000","last_peered":"2024-04-25T15:19:52.489205+0000","last_clean":"2024-04-25T15:19:52.489205+0000","last_became_active":"2024-04-25T15:19:46.486203+0000","last_became_peered":"2024-04-25T15:19:46.486203+0000","last_unstale":"2024-04-25T15:19:52.489205+0000","last_undegraded":"2024-04-25T15:19:52.489205+0000","last_fullsized":"2024-04-25T15:19:52.489205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490288+0000","last_change":"2024-04-25T15:19:50.510692+0000","last_active":"2024-04-25T15:19:52.490288+0000","last_peered":"2024-04-25T15:19:52.490288+0000","last_clean":"2024-04-25T15:19:52.490288+0000","last_became_active":"2024-04-25T15:19:50.510156+0000","last_became_peered":"2024-04-25T15:19:50.510156+0000","last_unstale":"2024-04-25T15:19:52.490288+0000","last_undegraded":"2024-04-25T15:19:52.490288+0000","last_fullsized":"2024-04-25T15:19:52.490288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396965+0000","last_change":"2024-04-25T15:19:45.471053+0000","last_active":"2024-04-25T15:19:53.396965+0000","last_peered":"2024-04-25T15:19:53.396965+0000","last_clean":"2024-04-25T15:19:53.396965+0000","last_became_active":"2024-04-25T15:19:45.470704+0000","last_became_peered":"2024-04-25T15:19:45.470704+0000","last_unstale":"2024-04-25T15:19:53.396965+0000","last_undegraded":"2024-04-25T15:19:53.396965+0000","last_fullsized":"2024-04-25T15:19:53.396965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:55.012973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491359+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.491359+0000","last_peered":"2024-04-25T15:19:52.491359+0000","last_clean":"2024-04-25T15:19:52.491359+0000","last_became_active":"2024-04-25T15:19:46.482181+0000","last_became_peered":"2024-04-25T15:19:46.482181+0000","last_unstale":"2024-04-25T15:19:52.491359+0000","last_undegraded":"2024-04-25T15:19:52.491359+0000","last_fullsized":"2024-04-25T15:19:52.491359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492195+0000","last_change":"2024-04-25T15:19:50.515950+0000","last_active":"2024-04-25T15:19:52.492195+0000","last_peered":"2024-04-25T15:19:52.492195+0000","last_clean":"2024-04-25T15:19:52.492195+0000","last_became_active":"2024-04-25T15:19:50.515486+0000","last_became_peered":"2024-04-25T15:19:50.515486+0000","last_unstale":"2024-04-25T15:19:52.492195+0000","last_undegraded":"2024-04-25T15:19:52.492195+0000","last_fullsized":"2024-04-25T15:19:52.492195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396633+0000","last_change":"2024-04-25T15:19:45.472051+0000","last_active":"2024-04-25T15:19:53.396633+0000","last_peered":"2024-04-25T15:19:53.396633+0000","last_clean":"2024-04-25T15:19:53.396633+0000","last_became_active":"2024-04-25T15:19:45.471710+0000","last_became_peered":"2024-04-25T15:19:45.471710+0000","last_unstale":"2024-04-25T15:19:53.396633+0000","last_undegraded":"2024-04-25T15:19:53.396633+0000","last_fullsized":"2024-04-25T15:19:53.396633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491729+0000","last_change":"2024-04-25T15:19:46.483085+0000","last_active":"2024-04-25T15:19:52.491729+0000","last_peered":"2024-04-25T15:19:52.491729+0000","last_clean":"2024-04-25T15:19:52.491729+0000","last_became_active":"2024-04-25T15:19:46.482749+0000","last_became_peered":"2024-04-25T15:19:46.482749+0000","last_unstale":"2024-04-25T15:19:52.491729+0000","last_undegraded":"2024-04-25T15:19:52.491729+0000","last_fullsized":"2024-04-25T15:19:52.491729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490380+0000","last_change":"2024-04-25T15:19:50.508661+0000","last_active":"2024-04-25T15:19:52.490380+0000","last_peered":"2024-04-25T15:19:52.490380+0000","last_clean":"2024-04-25T15:19:52.490380+0000","last_became_active":"2024-04-25T15:19:50.508116+0000","last_became_peered":"2024-04-25T15:19:50.508116+0000","last_unstale":"2024-04-25T15:19:52.490380+0000","last_undegraded":"2024-04-25T15:19:52.490380+0000","last_fullsized":"2024-04-25T15:19:52.490380+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018240+0000","last_change":"2024-04-25T15:19:46.477569+0000","last_active":"2024-04-25T15:19:54.018240+0000","last_peered":"2024-04-25T15:19:54.018240+0000","last_clean":"2024-04-25T15:19:54.018240+0000","last_became_active":"2024-04-25T15:19:46.477188+0000","last_became_peered":"2024-04-25T15:19:46.477188+0000","last_unstale":"2024-04-25T15:19:54.018240+0000","last_undegraded":"2024-04-25T15:19:54.018240+0000","last_fullsized":"2024-04-25T15:19:54.018240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396748+0000","last_change":"2024-04-25T15:19:45.469111+0000","last_active":"2024-04-25T15:19:53.396748+0000","last_peered":"2024-04-25T15:19:53.396748+0000","last_clean":"2024-04-25T15:19:53.396748+0000","last_became_active":"2024-04-25T15:19:45.468780+0000","last_became_peered":"2024-04-25T15:19:45.468780+0000","last_unstale":"2024-04-25T15:19:53.396748+0000","last_undegraded":"2024-04-25T15:19:53.396748+0000","last_fullsized":"2024-04-25T15:19:53.396748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491203+0000","last_change":"2024-04-25T15:19:50.513416+0000","last_active":"2024-04-25T15:19:52.491203+0000","last_peered":"2024-04-25T15:19:52.491203+0000","last_clean":"2024-04-25T15:19:52.491203+0000","last_became_active":"2024-04-25T15:19:50.513024+0000","last_became_peered":"2024-04-25T15:19:50.513024+0000","last_unstale":"2024-04-25T15:19:52.491203+0000","last_undegraded":"2024-04-25T15:19:52.491203+0000","last_fullsized":"2024-04-25T15:19:52.491203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:20.334921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020296+0000","last_change":"2024-04-25T15:19:46.472167+0000","last_active":"2024-04-25T15:19:54.020296+0000","last_peered":"2024-04-25T15:19:54.020296+0000","last_clean":"2024-04-25T15:19:54.020296+0000","last_became_active":"2024-04-25T15:19:46.471839+0000","last_became_peered":"2024-04-25T15:19:46.471839+0000","last_unstale":"2024-04-25T15:19:54.020296+0000","last_undegraded":"2024-04-25T15:19:54.020296+0000","last_fullsized":"2024-04-25T15:19:54.020296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809709+0000","last_change":"2024-04-25T15:19:45.467435+0000","last_active":"2024-04-25T15:19:53.809709+0000","last_peered":"2024-04-25T15:19:53.809709+0000","last_clean":"2024-04-25T15:19:53.809709+0000","last_became_active":"2024-04-25T15:19:45.466960+0000","last_became_peered":"2024-04-25T15:19:45.466960+0000","last_unstale":"2024-04-25T15:19:53.809709+0000","last_undegraded":"2024-04-25T15:19:53.809709+0000","last_fullsized":"2024-04-25T15:19:53.809709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:55.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491020+0000","last_change":"2024-04-25T15:19:50.515801+0000","last_active":"2024-04-25T15:19:52.491020+0000","last_peered":"2024-04-25T15:19:52.491020+0000","last_clean":"2024-04-25T15:19:52.491020+0000","last_became_active":"2024-04-25T15:19:50.515340+0000","last_became_peered":"2024-04-25T15:19:50.515340+0000","last_unstale":"2024-04-25T15:19:52.491020+0000","last_undegraded":"2024-04-25T15:19:52.491020+0000","last_fullsized":"2024-04-25T15:19:52.491020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:39.634911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020955+0000","last_change":"2024-04-25T15:19:46.470328+0000","last_active":"2024-04-25T15:19:54.020955+0000","last_peered":"2024-04-25T15:19:54.020955+0000","last_clean":"2024-04-25T15:19:54.020955+0000","last_became_active":"2024-04-25T15:19:46.469972+0000","last_became_peered":"2024-04-25T15:19:46.469972+0000","last_unstale":"2024-04-25T15:19:54.020955+0000","last_undegraded":"2024-04-25T15:19:54.020955+0000","last_fullsized":"2024-04-25T15:19:54.020955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.808963+0000","last_change":"2024-04-25T15:19:45.468754+0000","last_active":"2024-04-25T15:19:53.808963+0000","last_peered":"2024-04-25T15:19:53.808963+0000","last_clean":"2024-04-25T15:19:53.808963+0000","last_became_active":"2024-04-25T15:19:45.468374+0000","last_became_peered":"2024-04-25T15:19:45.468374+0000","last_unstale":"2024-04-25T15:19:53.808963+0000","last_undegraded":"2024-04-25T15:19:53.808963+0000","last_fullsized":"2024-04-25T15:19:53.808963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810131+0000","last_change":"2024-04-25T15:19:51.101484+0000","last_active":"2024-04-25T15:19:53.810131+0000","last_peered":"2024-04-25T15:19:53.810131+0000","last_clean":"2024-04-25T15:19:53.810131+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:53.810131+0000","last_undegraded":"2024-04-25T15:19:53.810131+0000","last_fullsized":"2024-04-25T15:19:53.810131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:57.500581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491979+0000","last_change":"2024-04-25T15:19:46.458762+0000","last_active":"2024-04-25T15:19:52.491979+0000","last_peered":"2024-04-25T15:19:52.491979+0000","last_clean":"2024-04-25T15:19:52.491979+0000","last_became_active":"2024-04-25T15:19:46.458438+0000","last_became_peered":"2024-04-25T15:19:46.458438+0000","last_unstale":"2024-04-25T15:19:52.491979+0000","last_undegraded":"2024-04-25T15:19:52.491979+0000","last_fullsized":"2024-04-25T15:19:52.491979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.402061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396261+0000","last_change":"2024-04-25T15:19:45.471672+0000","last_active":"2024-04-25T15:19:53.396261+0000","last_peered":"2024-04-25T15:19:53.396261+0000","last_clean":"2024-04-25T15:19:53.396261+0000","last_became_active":"2024-04-25T15:19:45.471095+0000","last_became_peered":"2024-04-25T15:19:45.471095+0000","last_unstale":"2024-04-25T15:19:53.396261+0000","last_undegraded":"2024-04-25T15:19:53.396261+0000","last_fullsized":"2024-04-25T15:19:53.396261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490630+0000","last_change":"2024-04-25T15:19:50.511923+0000","last_active":"2024-04-25T15:19:52.490630+0000","last_peered":"2024-04-25T15:19:52.490630+0000","last_clean":"2024-04-25T15:19:52.490630+0000","last_became_active":"2024-04-25T15:19:50.511350+0000","last_became_peered":"2024-04-25T15:19:50.511350+0000","last_unstale":"2024-04-25T15:19:52.490630+0000","last_undegraded":"2024-04-25T15:19:52.490630+0000","last_fullsized":"2024-04-25T15:19:52.490630+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489343+0000","last_change":"2024-04-25T15:19:46.472562+0000","last_active":"2024-04-25T15:19:52.489343+0000","last_peered":"2024-04-25T15:19:52.489343+0000","last_clean":"2024-04-25T15:19:52.489343+0000","last_became_active":"2024-04-25T15:19:46.471397+0000","last_became_peered":"2024-04-25T15:19:46.471397+0000","last_unstale":"2024-04-25T15:19:52.489343+0000","last_undegraded":"2024-04-25T15:19:52.489343+0000","last_fullsized":"2024-04-25T15:19:52.489343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017764+0000","last_change":"2024-04-25T15:19:45.467859+0000","last_active":"2024-04-25T15:19:54.017764+0000","last_peered":"2024-04-25T15:19:54.017764+0000","last_clean":"2024-04-25T15:19:54.017764+0000","last_became_active":"2024-04-25T15:19:45.467461+0000","last_became_peered":"2024-04-25T15:19:45.467461+0000","last_unstale":"2024-04-25T15:19:54.017764+0000","last_undegraded":"2024-04-25T15:19:54.017764+0000","last_fullsized":"2024-04-25T15:19:54.017764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491639+0000","last_change":"2024-04-25T15:19:51.101973+0000","last_active":"2024-04-25T15:19:52.491639+0000","last_peered":"2024-04-25T15:19:52.491639+0000","last_clean":"2024-04-25T15:19:52.491639+0000","last_became_active":"2024-04-25T15:19:51.101046+0000","last_became_peered":"2024-04-25T15:19:51.101046+0000","last_unstale":"2024-04-25T15:19:52.491639+0000","last_undegraded":"2024-04-25T15:19:52.491639+0000","last_fullsized":"2024-04-25T15:19:52.491639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020585+0000","last_change":"2024-04-25T15:19:46.481579+0000","last_active":"2024-04-25T15:19:54.020585+0000","last_peered":"2024-04-25T15:19:54.020585+0000","last_clean":"2024-04-25T15:19:54.020585+0000","last_became_active":"2024-04-25T15:19:46.481265+0000","last_became_peered":"2024-04-25T15:19:46.481265+0000","last_unstale":"2024-04-25T15:19:54.020585+0000","last_undegraded":"2024-04-25T15:19:54.020585+0000","last_fullsized":"2024-04-25T15:19:54.020585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810014+0000","last_change":"2024-04-25T15:19:45.464607+0000","last_active":"2024-04-25T15:19:53.810014+0000","last_peered":"2024-04-25T15:19:53.810014+0000","last_clean":"2024-04-25T15:19:53.810014+0000","last_became_active":"2024-04-25T15:19:45.464278+0000","last_became_peered":"2024-04-25T15:19:45.464278+0000","last_unstale":"2024-04-25T15:19:53.810014+0000","last_undegraded":"2024-04-25T15:19:53.810014+0000","last_fullsized":"2024-04-25T15:19:53.810014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020107+0000","last_change":"2024-04-25T15:19:51.101501+0000","last_active":"2024-04-25T15:19:54.020107+0000","last_peered":"2024-04-25T15:19:54.020107+0000","last_clean":"2024-04-25T15:19:54.020107+0000","last_became_active":"2024-04-25T15:19:51.101130+0000","last_became_peered":"2024-04-25T15:19:51.101130+0000","last_unstale":"2024-04-25T15:19:54.020107+0000","last_undegraded":"2024-04-25T15:19:54.020107+0000","last_fullsized":"2024-04-25T15:19:54.020107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809775+0000","last_change":"2024-04-25T15:19:46.469600+0000","last_active":"2024-04-25T15:19:53.809775+0000","last_peered":"2024-04-25T15:19:53.809775+0000","last_clean":"2024-04-25T15:19:53.809775+0000","last_became_active":"2024-04-25T15:19:46.469254+0000","last_became_peered":"2024-04-25T15:19:46.469254+0000","last_unstale":"2024-04-25T15:19:53.809775+0000","last_undegraded":"2024-04-25T15:19:53.809775+0000","last_fullsized":"2024-04-25T15:19:53.809775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397107+0000","last_change":"2024-04-25T15:19:45.471728+0000","last_active":"2024-04-25T15:19:53.397107+0000","last_peered":"2024-04-25T15:19:53.397107+0000","last_clean":"2024-04-25T15:19:53.397107+0000","last_became_active":"2024-04-25T15:19:45.471426+0000","last_became_peered":"2024-04-25T15:19:45.471426+0000","last_unstale":"2024-04-25T15:19:53.397107+0000","last_undegraded":"2024-04-25T15:19:53.397107+0000","last_fullsized":"2024-04-25T15:19:53.397107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:39:45.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488880+0000","last_change":"2024-04-25T15:19:51.101520+0000","last_active":"2024-04-25T15:19:52.488880+0000","last_peered":"2024-04-25T15:19:52.488880+0000","last_clean":"2024-04-25T15:19:52.488880+0000","last_became_active":"2024-04-25T15:19:51.100989+0000","last_became_peered":"2024-04-25T15:19:51.100989+0000","last_unstale":"2024-04-25T15:19:52.488880+0000","last_undegraded":"2024-04-25T15:19:52.488880+0000","last_fullsized":"2024-04-25T15:19:52.488880+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:04.999869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491850+0000","last_change":"2024-04-25T15:19:46.482591+0000","last_active":"2024-04-25T15:19:52.491850+0000","last_peered":"2024-04-25T15:19:52.491850+0000","last_clean":"2024-04-25T15:19:52.491850+0000","last_became_active":"2024-04-25T15:19:46.482265+0000","last_became_peered":"2024-04-25T15:19:46.482265+0000","last_unstale":"2024-04-25T15:19:52.491850+0000","last_undegraded":"2024-04-25T15:19:52.491850+0000","last_fullsized":"2024-04-25T15:19:52.491850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809000+0000","last_change":"2024-04-25T15:19:45.470984+0000","last_active":"2024-04-25T15:19:53.809000+0000","last_peered":"2024-04-25T15:19:53.809000+0000","last_clean":"2024-04-25T15:19:53.809000+0000","last_became_active":"2024-04-25T15:19:45.470665+0000","last_became_peered":"2024-04-25T15:19:45.470665+0000","last_unstale":"2024-04-25T15:19:53.809000+0000","last_undegraded":"2024-04-25T15:19:53.809000+0000","last_fullsized":"2024-04-25T15:19:53.809000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809497+0000","last_change":"2024-04-25T15:19:50.509138+0000","last_active":"2024-04-25T15:19:53.809497+0000","last_peered":"2024-04-25T15:19:53.809497+0000","last_clean":"2024-04-25T15:19:53.809497+0000","last_became_active":"2024-04-25T15:19:50.508713+0000","last_became_peered":"2024-04-25T15:19:50.508713+0000","last_unstale":"2024-04-25T15:19:53.809497+0000","last_undegraded":"2024-04-25T15:19:53.809497+0000","last_fullsized":"2024-04-25T15:19:53.809497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492266+0000","last_change":"2024-04-25T15:19:46.459065+0000","last_active":"2024-04-25T15:19:52.492266+0000","last_peered":"2024-04-25T15:19:52.492266+0000","last_clean":"2024-04-25T15:19:52.492266+0000","last_became_active":"2024-04-25T15:19:46.458708+0000","last_became_peered":"2024-04-25T15:19:46.458708+0000","last_unstale":"2024-04-25T15:19:52.492266+0000","last_undegraded":"2024-04-25T15:19:52.492266+0000","last_fullsized":"2024-04-25T15:19:52.492266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:16.345566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491129+0000","last_change":"2024-04-25T15:19:45.464370+0000","last_active":"2024-04-25T15:19:52.491129+0000","last_peered":"2024-04-25T15:19:52.491129+0000","last_clean":"2024-04-25T15:19:52.491129+0000","last_became_active":"2024-04-25T15:19:45.464041+0000","last_became_peered":"2024-04-25T15:19:45.464041+0000","last_unstale":"2024-04-25T15:19:52.491129+0000","last_undegraded":"2024-04-25T15:19:52.491129+0000","last_fullsized":"2024-04-25T15:19:52.491129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809793+0000","last_change":"2024-04-25T15:19:50.516748+0000","last_active":"2024-04-25T15:19:53.809793+0000","last_peered":"2024-04-25T15:19:53.809793+0000","last_clean":"2024-04-25T15:19:53.809793+0000","last_became_active":"2024-04-25T15:19:50.515480+0000","last_became_peered":"2024-04-25T15:19:50.515480+0000","last_unstale":"2024-04-25T15:19:53.809793+0000","last_undegraded":"2024-04-25T15:19:53.809793+0000","last_fullsized":"2024-04-25T15:19:53.809793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:54.507021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809413+0000","last_change":"2024-04-25T15:19:46.469618+0000","last_active":"2024-04-25T15:19:53.809413+0000","last_peered":"2024-04-25T15:19:53.809413+0000","last_clean":"2024-04-25T15:19:53.809413+0000","last_became_active":"2024-04-25T15:19:46.469289+0000","last_became_peered":"2024-04-25T15:19:46.469289+0000","last_unstale":"2024-04-25T15:19:53.809413+0000","last_undegraded":"2024-04-25T15:19:53.809413+0000","last_fullsized":"2024-04-25T15:19:53.809413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020301+0000","last_change":"2024-04-25T15:19:45.462951+0000","last_active":"2024-04-25T15:19:54.020301+0000","last_peered":"2024-04-25T15:19:54.020301+0000","last_clean":"2024-04-25T15:19:54.020301+0000","last_became_active":"2024-04-25T15:19:45.462424+0000","last_became_peered":"2024-04-25T15:19:45.462424+0000","last_unstale":"2024-04-25T15:19:54.020301+0000","last_undegraded":"2024-04-25T15:19:54.020301+0000","last_fullsized":"2024-04-25T15:19:54.020301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.810177+0000","last_change":"2024-04-25T15:19:50.519794+0000","last_active":"2024-04-25T15:19:53.810177+0000","last_peered":"2024-04-25T15:19:53.810177+0000","last_clean":"2024-04-25T15:19:53.810177+0000","last_became_active":"2024-04-25T15:19:50.519278+0000","last_became_peered":"2024-04-25T15:19:50.519278+0000","last_unstale":"2024-04-25T15:19:53.810177+0000","last_undegraded":"2024-04-25T15:19:53.810177+0000","last_fullsized":"2024-04-25T15:19:53.810177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:14.349158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488686+0000","last_change":"2024-04-25T15:19:46.459839+0000","last_active":"2024-04-25T15:19:52.488686+0000","last_peered":"2024-04-25T15:19:52.488686+0000","last_clean":"2024-04-25T15:19:52.488686+0000","last_became_active":"2024-04-25T15:19:46.459401+0000","last_became_peered":"2024-04-25T15:19:46.459401+0000","last_unstale":"2024-04-25T15:19:52.488686+0000","last_undegraded":"2024-04-25T15:19:52.488686+0000","last_fullsized":"2024-04-25T15:19:52.488686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:46.303899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396291+0000","last_change":"2024-04-25T15:19:45.471412+0000","last_active":"2024-04-25T15:19:53.396291+0000","last_peered":"2024-04-25T15:19:53.396291+0000","last_clean":"2024-04-25T15:19:53.396291+0000","last_became_active":"2024-04-25T15:19:45.470632+0000","last_became_peered":"2024-04-25T15:19:45.470632+0000","last_unstale":"2024-04-25T15:19:53.396291+0000","last_undegraded":"2024-04-25T15:19:53.396291+0000","last_fullsized":"2024-04-25T15:19:53.396291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489312+0000","last_change":"2024-04-25T15:19:51.101653+0000","last_active":"2024-04-25T15:19:52.489312+0000","last_peered":"2024-04-25T15:19:52.489312+0000","last_clean":"2024-04-25T15:19:52.489312+0000","last_became_active":"2024-04-25T15:19:51.101246+0000","last_became_peered":"2024-04-25T15:19:51.101246+0000","last_unstale":"2024-04-25T15:19:52.489312+0000","last_undegraded":"2024-04-25T15:19:52.489312+0000","last_fullsized":"2024-04-25T15:19:52.489312+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020754+0000","last_change":"2024-04-25T15:19:46.488866+0000","last_active":"2024-04-25T15:19:54.020754+0000","last_peered":"2024-04-25T15:19:54.020754+0000","last_clean":"2024-04-25T15:19:54.020754+0000","last_became_active":"2024-04-25T15:19:46.488494+0000","last_became_peered":"2024-04-25T15:19:46.488494+0000","last_unstale":"2024-04-25T15:19:54.020754+0000","last_undegraded":"2024-04-25T15:19:54.020754+0000","last_fullsized":"2024-04-25T15:19:54.020754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490534+0000","last_change":"2024-04-25T15:19:45.459379+0000","last_active":"2024-04-25T15:19:52.490534+0000","last_peered":"2024-04-25T15:19:52.490534+0000","last_clean":"2024-04-25T15:19:52.490534+0000","last_became_active":"2024-04-25T15:19:45.458985+0000","last_became_peered":"2024-04-25T15:19:45.458985+0000","last_unstale":"2024-04-25T15:19:52.490534+0000","last_undegraded":"2024-04-25T15:19:52.490534+0000","last_fullsized":"2024-04-25T15:19:52.490534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396508+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:19:53.396508+0000","last_peered":"2024-04-25T15:19:53.396508+0000","last_clean":"2024-04-25T15:19:53.396508+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:19:53.396508+0000","last_undegraded":"2024-04-25T15:19:53.396508+0000","last_fullsized":"2024-04-25T15:19:53.396508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018620+0000","last_change":"2024-04-25T15:19:51.102362+0000","last_active":"2024-04-25T15:19:54.018620+0000","last_peered":"2024-04-25T15:19:54.018620+0000","last_clean":"2024-04-25T15:19:54.018620+0000","last_became_active":"2024-04-25T15:19:51.101137+0000","last_became_peered":"2024-04-25T15:19:51.101137+0000","last_unstale":"2024-04-25T15:19:54.018620+0000","last_undegraded":"2024-04-25T15:19:54.018620+0000","last_fullsized":"2024-04-25T15:19:54.018620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:47.107741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396808+0000","last_change":"2024-04-25T15:19:46.486533+0000","last_active":"2024-04-25T15:19:53.396808+0000","last_peered":"2024-04-25T15:19:53.396808+0000","last_clean":"2024-04-25T15:19:53.396808+0000","last_became_active":"2024-04-25T15:19:46.485775+0000","last_became_peered":"2024-04-25T15:19:46.485775+0000","last_unstale":"2024-04-25T15:19:53.396808+0000","last_undegraded":"2024-04-25T15:19:53.396808+0000","last_fullsized":"2024-04-25T15:19:53.396808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019912+0000","last_change":"2024-04-25T15:19:45.467231+0000","last_active":"2024-04-25T15:19:54.019912+0000","last_peered":"2024-04-25T15:19:54.019912+0000","last_clean":"2024-04-25T15:19:54.019912+0000","last_became_active":"2024-04-25T15:19:45.466874+0000","last_became_peered":"2024-04-25T15:19:45.466874+0000","last_unstale":"2024-04-25T15:19:54.019912+0000","last_undegraded":"2024-04-25T15:19:54.019912+0000","last_fullsized":"2024-04-25T15:19:54.019912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018302+0000","last_change":"2024-04-25T15:19:51.101630+0000","last_active":"2024-04-25T15:19:54.018302+0000","last_peered":"2024-04-25T15:19:54.018302+0000","last_clean":"2024-04-25T15:19:54.018302+0000","last_became_active":"2024-04-25T15:19:51.101121+0000","last_became_peered":"2024-04-25T15:19:51.101121+0000","last_unstale":"2024-04-25T15:19:54.018302+0000","last_undegraded":"2024-04-25T15:19:54.018302+0000","last_fullsized":"2024-04-25T15:19:54.018302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:45.364124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491492+0000","last_change":"2024-04-25T15:19:46.483183+0000","last_active":"2024-04-25T15:19:52.491492+0000","last_peered":"2024-04-25T15:19:52.491492+0000","last_clean":"2024-04-25T15:19:52.491492+0000","last_became_active":"2024-04-25T15:19:46.482618+0000","last_became_peered":"2024-04-25T15:19:46.482618+0000","last_unstale":"2024-04-25T15:19:52.491492+0000","last_undegraded":"2024-04-25T15:19:52.491492+0000","last_fullsized":"2024-04-25T15:19:52.491492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809024+0000","last_change":"2024-04-25T15:19:45.469018+0000","last_active":"2024-04-25T15:19:53.809024+0000","last_peered":"2024-04-25T15:19:53.809024+0000","last_clean":"2024-04-25T15:19:53.809024+0000","last_became_active":"2024-04-25T15:19:45.468712+0000","last_became_peered":"2024-04-25T15:19:45.468712+0000","last_unstale":"2024-04-25T15:19:53.809024+0000","last_undegraded":"2024-04-25T15:19:53.809024+0000","last_fullsized":"2024-04-25T15:19:53.809024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:23.965251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489037+0000","last_change":"2024-04-25T15:19:50.513573+0000","last_active":"2024-04-25T15:19:52.489037+0000","last_peered":"2024-04-25T15:19:52.489037+0000","last_clean":"2024-04-25T15:19:52.489037+0000","last_became_active":"2024-04-25T15:19:50.513018+0000","last_became_peered":"2024-04-25T15:19:50.513018+0000","last_unstale":"2024-04-25T15:19:52.489037+0000","last_undegraded":"2024-04-25T15:19:52.489037+0000","last_fullsized":"2024-04-25T15:19:52.489037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397172+0000","last_change":"2024-04-25T15:19:46.465000+0000","last_active":"2024-04-25T15:19:53.397172+0000","last_peered":"2024-04-25T15:19:53.397172+0000","last_clean":"2024-04-25T15:19:53.397172+0000","last_became_active":"2024-04-25T15:19:46.464680+0000","last_became_peered":"2024-04-25T15:19:46.464680+0000","last_unstale":"2024-04-25T15:19:53.397172+0000","last_undegraded":"2024-04-25T15:19:53.397172+0000","last_fullsized":"2024-04-25T15:19:53.397172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:35.597757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020764+0000","last_change":"2024-04-25T15:19:45.463067+0000","last_active":"2024-04-25T15:19:54.020764+0000","last_peered":"2024-04-25T15:19:54.020764+0000","last_clean":"2024-04-25T15:19:54.020764+0000","last_became_active":"2024-04-25T15:19:45.462611+0000","last_became_peered":"2024-04-25T15:19:45.462611+0000","last_unstale":"2024-04-25T15:19:54.020764+0000","last_undegraded":"2024-04-25T15:19:54.020764+0000","last_fullsized":"2024-04-25T15:19:54.020764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490099+0000","last_change":"2024-04-25T15:19:51.101470+0000","last_active":"2024-04-25T15:19:52.490099+0000","last_peered":"2024-04-25T15:19:52.490099+0000","last_clean":"2024-04-25T15:19:52.490099+0000","last_became_active":"2024-04-25T15:19:51.100991+0000","last_became_peered":"2024-04-25T15:19:51.100991+0000","last_unstale":"2024-04-25T15:19:52.490099+0000","last_undegraded":"2024-04-25T15:19:52.490099+0000","last_fullsized":"2024-04-25T15:19:52.490099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396886+0000","last_change":"2024-04-25T15:19:46.478228+0000","last_active":"2024-04-25T15:19:53.396886+0000","last_peered":"2024-04-25T15:19:53.396886+0000","last_clean":"2024-04-25T15:19:53.396886+0000","last_became_active":"2024-04-25T15:19:46.477939+0000","last_became_peered":"2024-04-25T15:19:46.477939+0000","last_unstale":"2024-04-25T15:19:53.396886+0000","last_undegraded":"2024-04-25T15:19:53.396886+0000","last_fullsized":"2024-04-25T15:19:53.396886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:18.173926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491269+0000","last_change":"2024-04-25T15:19:45.456572+0000","last_active":"2024-04-25T15:19:52.491269+0000","last_peered":"2024-04-25T15:19:52.491269+0000","last_clean":"2024-04-25T15:19:52.491269+0000","last_became_active":"2024-04-25T15:19:45.456238+0000","last_became_peered":"2024-04-25T15:19:45.456238+0000","last_unstale":"2024-04-25T15:19:52.491269+0000","last_undegraded":"2024-04-25T15:19:52.491269+0000","last_fullsized":"2024-04-25T15:19:52.491269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:09.326979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491720+0000","last_change":"2024-04-25T15:19:50.511879+0000","last_active":"2024-04-25T15:19:52.491720+0000","last_peered":"2024-04-25T15:19:52.491720+0000","last_clean":"2024-04-25T15:19:52.491720+0000","last_became_active":"2024-04-25T15:19:50.511380+0000","last_became_peered":"2024-04-25T15:19:50.511380+0000","last_unstale":"2024-04-25T15:19:52.491720+0000","last_undegraded":"2024-04-25T15:19:52.491720+0000","last_fullsized":"2024-04-25T15:19:52.491720+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:32.855448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396446+0000","last_change":"2024-04-25T15:19:46.465026+0000","last_active":"2024-04-25T15:19:53.396446+0000","last_peered":"2024-04-25T15:19:53.396446+0000","last_clean":"2024-04-25T15:19:53.396446+0000","last_became_active":"2024-04-25T15:19:46.464703+0000","last_became_peered":"2024-04-25T15:19:46.464703+0000","last_unstale":"2024-04-25T15:19:53.396446+0000","last_undegraded":"2024-04-25T15:19:53.396446+0000","last_fullsized":"2024-04-25T15:19:53.396446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:16.504035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020165+0000","last_change":"2024-04-25T15:19:45.465017+0000","last_active":"2024-04-25T15:19:54.020165+0000","last_peered":"2024-04-25T15:19:54.020165+0000","last_clean":"2024-04-25T15:19:54.020165+0000","last_became_active":"2024-04-25T15:19:45.464691+0000","last_became_peered":"2024-04-25T15:19:45.464691+0000","last_unstale":"2024-04-25T15:19:54.020165+0000","last_undegraded":"2024-04-25T15:19:54.020165+0000","last_fullsized":"2024-04-25T15:19:54.020165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489222+0000","last_change":"2024-04-25T15:19:50.508942+0000","last_active":"2024-04-25T15:19:52.489222+0000","last_peered":"2024-04-25T15:19:52.489222+0000","last_clean":"2024-04-25T15:19:52.489222+0000","last_became_active":"2024-04-25T15:19:50.508482+0000","last_became_peered":"2024-04-25T15:19:50.508482+0000","last_unstale":"2024-04-25T15:19:52.489222+0000","last_undegraded":"2024-04-25T15:19:52.489222+0000","last_fullsized":"2024-04-25T15:19:52.489222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:17.965669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018243+0000","last_change":"2024-04-25T15:19:46.473708+0000","last_active":"2024-04-25T15:19:54.018243+0000","last_peered":"2024-04-25T15:19:54.018243+0000","last_clean":"2024-04-25T15:19:54.018243+0000","last_became_active":"2024-04-25T15:19:46.473372+0000","last_became_peered":"2024-04-25T15:19:46.473372+0000","last_unstale":"2024-04-25T15:19:54.018243+0000","last_undegraded":"2024-04-25T15:19:54.018243+0000","last_fullsized":"2024-04-25T15:19:54.018243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.469745+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.469458+0000","last_became_peered":"2024-04-25T15:19:45.469458+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:36.957490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020364+0000","last_change":"2024-04-25T15:19:51.101455+0000","last_active":"2024-04-25T15:19:54.020364+0000","last_peered":"2024-04-25T15:19:54.020364+0000","last_clean":"2024-04-25T15:19:54.020364+0000","last_became_active":"2024-04-25T15:19:51.101028+0000","last_became_peered":"2024-04-25T15:19:51.101028+0000","last_unstale":"2024-04-25T15:19:54.020364+0000","last_undegraded":"2024-04-25T15:19:54.020364+0000","last_fullsized":"2024-04-25T15:19:54.020364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:05.089655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396638+0000","last_change":"2024-04-25T15:19:46.465773+0000","last_active":"2024-04-25T15:19:53.396638+0000","last_peered":"2024-04-25T15:19:53.396638+0000","last_clean":"2024-04-25T15:19:53.396638+0000","last_became_active":"2024-04-25T15:19:46.465467+0000","last_became_peered":"2024-04-25T15:19:46.465467+0000","last_unstale":"2024-04-25T15:19:53.396638+0000","last_undegraded":"2024-04-25T15:19:53.396638+0000","last_fullsized":"2024-04-25T15:19:53.396638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491007+0000","last_change":"2024-04-25T15:19:45.463534+0000","last_active":"2024-04-25T15:19:52.491007+0000","last_peered":"2024-04-25T15:19:52.491007+0000","last_clean":"2024-04-25T15:19:52.491007+0000","last_became_active":"2024-04-25T15:19:45.463167+0000","last_became_peered":"2024-04-25T15:19:45.463167+0000","last_unstale":"2024-04-25T15:19:52.491007+0000","last_undegraded":"2024-04-25T15:19:52.491007+0000","last_fullsized":"2024-04-25T15:19:52.491007+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:56.950845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021379+0000","last_change":"2024-04-25T15:19:51.102717+0000","last_active":"2024-04-25T15:19:54.021379+0000","last_peered":"2024-04-25T15:19:54.021379+0000","last_clean":"2024-04-25T15:19:54.021379+0000","last_became_active":"2024-04-25T15:19:51.102247+0000","last_became_peered":"2024-04-25T15:19:51.102247+0000","last_unstale":"2024-04-25T15:19:54.021379+0000","last_undegraded":"2024-04-25T15:19:54.021379+0000","last_fullsized":"2024-04-25T15:19:54.021379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:14:00.372742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489038+0000","last_change":"2024-04-25T15:19:46.460776+0000","last_active":"2024-04-25T15:19:52.489038+0000","last_peered":"2024-04-25T15:19:52.489038+0000","last_clean":"2024-04-25T15:19:52.489038+0000","last_became_active":"2024-04-25T15:19:46.460453+0000","last_became_peered":"2024-04-25T15:19:46.460453+0000","last_unstale":"2024-04-25T15:19:52.489038+0000","last_undegraded":"2024-04-25T15:19:52.489038+0000","last_fullsized":"2024-04-25T15:19:52.489038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489102+0000","last_change":"2024-04-25T15:19:45.462749+0000","last_active":"2024-04-25T15:19:52.489102+0000","last_peered":"2024-04-25T15:19:52.489102+0000","last_clean":"2024-04-25T15:19:52.489102+0000","last_became_active":"2024-04-25T15:19:45.462417+0000","last_became_peered":"2024-04-25T15:19:45.462417+0000","last_unstale":"2024-04-25T15:19:52.489102+0000","last_undegraded":"2024-04-25T15:19:52.489102+0000","last_fullsized":"2024-04-25T15:19:52.489102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491884+0000","last_change":"2024-04-25T15:19:51.102630+0000","last_active":"2024-04-25T15:19:52.491884+0000","last_peered":"2024-04-25T15:19:52.491884+0000","last_clean":"2024-04-25T15:19:52.491884+0000","last_became_active":"2024-04-25T15:19:51.101912+0000","last_became_peered":"2024-04-25T15:19:51.101912+0000","last_unstale":"2024-04-25T15:19:52.491884+0000","last_undegraded":"2024-04-25T15:19:52.491884+0000","last_fullsized":"2024-04-25T15:19:52.491884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396712+0000","last_change":"2024-04-25T15:19:46.471583+0000","last_active":"2024-04-25T15:19:53.396712+0000","last_peered":"2024-04-25T15:19:53.396712+0000","last_clean":"2024-04-25T15:19:53.396712+0000","last_became_active":"2024-04-25T15:19:46.471286+0000","last_became_peered":"2024-04-25T15:19:46.471286+0000","last_unstale":"2024-04-25T15:19:53.396712+0000","last_undegraded":"2024-04-25T15:19:53.396712+0000","last_fullsized":"2024-04-25T15:19:53.396712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:55.948605+0000","last_change":"2024-04-25T15:19:45.464680+0000","last_active":"2024-04-25T15:19:55.948605+0000","last_peered":"2024-04-25T15:19:55.948605+0000","last_clean":"2024-04-25T15:19:55.948605+0000","last_became_active":"2024-04-25T15:19:45.463690+0000","last_became_peered":"2024-04-25T15:19:45.463690+0000","last_unstale":"2024-04-25T15:19:55.948605+0000","last_undegraded":"2024-04-25T15:19:55.948605+0000","last_fullsized":"2024-04-25T15:19:55.948605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:14.428971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809637+0000","last_change":"2024-04-25T15:19:50.509907+0000","last_active":"2024-04-25T15:19:53.809637+0000","last_peered":"2024-04-25T15:19:53.809637+0000","last_clean":"2024-04-25T15:19:53.809637+0000","last_became_active":"2024-04-25T15:19:50.508793+0000","last_became_peered":"2024-04-25T15:19:50.508793+0000","last_unstale":"2024-04-25T15:19:53.809637+0000","last_undegraded":"2024-04-25T15:19:53.809637+0000","last_fullsized":"2024-04-25T15:19:53.809637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:03.764733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490746+0000","last_change":"2024-04-25T15:19:46.471815+0000","last_active":"2024-04-25T15:19:52.490746+0000","last_peered":"2024-04-25T15:19:52.490746+0000","last_clean":"2024-04-25T15:19:52.490746+0000","last_became_active":"2024-04-25T15:19:46.471473+0000","last_became_peered":"2024-04-25T15:19:46.471473+0000","last_unstale":"2024-04-25T15:19:52.490746+0000","last_undegraded":"2024-04-25T15:19:52.490746+0000","last_fullsized":"2024-04-25T15:19:52.490746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809045+0000","last_change":"2024-04-25T15:19:45.472821+0000","last_active":"2024-04-25T15:19:53.809045+0000","last_peered":"2024-04-25T15:19:53.809045+0000","last_clean":"2024-04-25T15:19:53.809045+0000","last_became_active":"2024-04-25T15:19:45.472420+0000","last_became_peered":"2024-04-25T15:19:45.472420+0000","last_unstale":"2024-04-25T15:19:53.809045+0000","last_undegraded":"2024-04-25T15:19:53.809045+0000","last_fullsized":"2024-04-25T15:19:53.809045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:29.303777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.492136+0000","last_change":"2024-04-25T15:19:50.509990+0000","last_active":"2024-04-25T15:19:52.492136+0000","last_peered":"2024-04-25T15:19:52.492136+0000","last_clean":"2024-04-25T15:19:52.492136+0000","last_became_active":"2024-04-25T15:19:50.509340+0000","last_became_peered":"2024-04-25T15:19:50.509340+0000","last_unstale":"2024-04-25T15:19:52.492136+0000","last_undegraded":"2024-04-25T15:19:52.492136+0000","last_fullsized":"2024-04-25T15:19:52.492136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489275+0000","last_change":"2024-04-25T15:19:46.485819+0000","last_active":"2024-04-25T15:19:52.489275+0000","last_peered":"2024-04-25T15:19:52.489275+0000","last_clean":"2024-04-25T15:19:52.489275+0000","last_became_active":"2024-04-25T15:19:46.485430+0000","last_became_peered":"2024-04-25T15:19:46.485430+0000","last_unstale":"2024-04-25T15:19:52.489275+0000","last_undegraded":"2024-04-25T15:19:52.489275+0000","last_fullsized":"2024-04-25T15:19:52.489275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490843+0000","last_change":"2024-04-25T15:19:45.461658+0000","last_active":"2024-04-25T15:19:52.490843+0000","last_peered":"2024-04-25T15:19:52.490843+0000","last_clean":"2024-04-25T15:19:52.490843+0000","last_became_active":"2024-04-25T15:19:45.461346+0000","last_became_peered":"2024-04-25T15:19:45.461346+0000","last_unstale":"2024-04-25T15:19:52.490843+0000","last_undegraded":"2024-04-25T15:19:52.490843+0000","last_fullsized":"2024-04-25T15:19:52.490843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.339605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021048+0000","last_change":"2024-04-25T15:19:50.518359+0000","last_active":"2024-04-25T15:19:54.021048+0000","last_peered":"2024-04-25T15:19:54.021048+0000","last_clean":"2024-04-25T15:19:54.021048+0000","last_became_active":"2024-04-25T15:19:50.517856+0000","last_became_peered":"2024-04-25T15:19:50.517856+0000","last_unstale":"2024-04-25T15:19:54.021048+0000","last_undegraded":"2024-04-25T15:19:54.021048+0000","last_fullsized":"2024-04-25T15:19:54.021048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:08:18.271358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490731+0000","last_change":"2024-04-25T15:19:46.469634+0000","last_active":"2024-04-25T15:19:52.490731+0000","last_peered":"2024-04-25T15:19:52.490731+0000","last_clean":"2024-04-25T15:19:52.490731+0000","last_became_active":"2024-04-25T15:19:46.468979+0000","last_became_peered":"2024-04-25T15:19:46.468979+0000","last_unstale":"2024-04-25T15:19:52.490731+0000","last_undegraded":"2024-04-25T15:19:52.490731+0000","last_fullsized":"2024-04-25T15:19:52.490731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:41.326969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017779+0000","last_change":"2024-04-25T15:19:45.471612+0000","last_active":"2024-04-25T15:19:54.017779+0000","last_peered":"2024-04-25T15:19:54.017779+0000","last_clean":"2024-04-25T15:19:54.017779+0000","last_became_active":"2024-04-25T15:19:45.471289+0000","last_became_peered":"2024-04-25T15:19:45.471289+0000","last_unstale":"2024-04-25T15:19:54.017779+0000","last_undegraded":"2024-04-25T15:19:54.017779+0000","last_fullsized":"2024-04-25T15:19:54.017779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020544+0000","last_change":"2024-04-25T15:19:51.105778+0000","last_active":"2024-04-25T15:19:54.020544+0000","last_peered":"2024-04-25T15:19:54.020544+0000","last_clean":"2024-04-25T15:19:54.020544+0000","last_became_active":"2024-04-25T15:19:51.105435+0000","last_became_peered":"2024-04-25T15:19:51.105435+0000","last_unstale":"2024-04-25T15:19:54.020544+0000","last_undegraded":"2024-04-25T15:19:54.020544+0000","last_fullsized":"2024-04-25T15:19:54.020544+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:09:15.216797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489181+0000","last_change":"2024-04-25T15:19:51.101540+0000","last_active":"2024-04-25T15:19:52.489181+0000","last_peered":"2024-04-25T15:19:52.489181+0000","last_clean":"2024-04-25T15:19:52.489181+0000","last_became_active":"2024-04-25T15:19:51.101126+0000","last_became_peered":"2024-04-25T15:19:51.101126+0000","last_unstale":"2024-04-25T15:19:52.489181+0000","last_undegraded":"2024-04-25T15:19:52.489181+0000","last_fullsized":"2024-04-25T15:19:52.489181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:09.633563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809353+0000","last_change":"2024-04-25T15:19:46.470144+0000","last_active":"2024-04-25T15:19:53.809353+0000","last_peered":"2024-04-25T15:19:53.809353+0000","last_clean":"2024-04-25T15:19:53.809353+0000","last_became_active":"2024-04-25T15:19:46.469786+0000","last_became_peered":"2024-04-25T15:19:46.469786+0000","last_unstale":"2024-04-25T15:19:53.809353+0000","last_undegraded":"2024-04-25T15:19:53.809353+0000","last_fullsized":"2024-04-25T15:19:53.809353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490304+0000","last_change":"2024-04-25T15:19:45.466078+0000","last_active":"2024-04-25T15:19:52.490304+0000","last_peered":"2024-04-25T15:19:52.490304+0000","last_clean":"2024-04-25T15:19:52.490304+0000","last_became_active":"2024-04-25T15:19:45.465744+0000","last_became_peered":"2024-04-25T15:19:45.465744+0000","last_unstale":"2024-04-25T15:19:52.490304+0000","last_undegraded":"2024-04-25T15:19:52.490304+0000","last_fullsized":"2024-04-25T15:19:52.490304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:46.864878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018744+0000","last_change":"2024-04-25T15:19:50.517045+0000","last_active":"2024-04-25T15:19:54.018744+0000","last_peered":"2024-04-25T15:19:54.018744+0000","last_clean":"2024-04-25T15:19:54.018744+0000","last_became_active":"2024-04-25T15:19:50.516571+0000","last_became_peered":"2024-04-25T15:19:50.516571+0000","last_unstale":"2024-04-25T15:19:54.018744+0000","last_undegraded":"2024-04-25T15:19:54.018744+0000","last_fullsized":"2024-04-25T15:19:54.018744+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:02.565310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020866+0000","last_change":"2024-04-25T15:19:46.472281+0000","last_active":"2024-04-25T15:19:54.020866+0000","last_peered":"2024-04-25T15:19:54.020866+0000","last_clean":"2024-04-25T15:19:54.020866+0000","last_became_active":"2024-04-25T15:19:46.471957+0000","last_became_peered":"2024-04-25T15:19:46.471957+0000","last_unstale":"2024-04-25T15:19:54.020866+0000","last_undegraded":"2024-04-25T15:19:54.020866+0000","last_fullsized":"2024-04-25T15:19:54.020866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.592246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021072+0000","last_change":"2024-04-25T15:19:45.461678+0000","last_active":"2024-04-25T15:19:54.021072+0000","last_peered":"2024-04-25T15:19:54.021072+0000","last_clean":"2024-04-25T15:19:54.021072+0000","last_became_active":"2024-04-25T15:19:45.461365+0000","last_became_peered":"2024-04-25T15:19:45.461365+0000","last_unstale":"2024-04-25T15:19:54.021072+0000","last_undegraded":"2024-04-25T15:19:54.021072+0000","last_fullsized":"2024-04-25T15:19:54.021072+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:19.401116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.017705+0000","last_change":"2024-04-25T15:19:45.468983+0000","last_active":"2024-04-25T15:19:54.017705+0000","last_peered":"2024-04-25T15:19:54.017705+0000","last_clean":"2024-04-25T15:19:54.017705+0000","last_became_active":"2024-04-25T15:19:45.468673+0000","last_became_peered":"2024-04-25T15:19:45.468673+0000","last_unstale":"2024-04-25T15:19:54.017705+0000","last_undegraded":"2024-04-25T15:19:54.017705+0000","last_fullsized":"2024-04-25T15:19:54.017705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:57.396099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489044+0000","last_change":"2024-04-25T15:19:46.459915+0000","last_active":"2024-04-25T15:19:52.489044+0000","last_peered":"2024-04-25T15:19:52.489044+0000","last_clean":"2024-04-25T15:19:52.489044+0000","last_became_active":"2024-04-25T15:19:46.459596+0000","last_became_peered":"2024-04-25T15:19:46.459596+0000","last_unstale":"2024-04-25T15:19:52.489044+0000","last_undegraded":"2024-04-25T15:19:52.489044+0000","last_fullsized":"2024-04-25T15:19:52.489044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020992+0000","last_change":"2024-04-25T15:19:50.517033+0000","last_active":"2024-04-25T15:19:54.020992+0000","last_peered":"2024-04-25T15:19:54.020992+0000","last_clean":"2024-04-25T15:19:54.020992+0000","last_became_active":"2024-04-25T15:19:50.516587+0000","last_became_peered":"2024-04-25T15:19:50.516587+0000","last_unstale":"2024-04-25T15:19:54.020992+0000","last_undegraded":"2024-04-25T15:19:54.020992+0000","last_fullsized":"2024-04-25T15:19:54.020992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:50.341053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.019861+0000","last_change":"2024-04-25T15:19:45.466373+0000","last_active":"2024-04-25T15:19:54.019861+0000","last_peered":"2024-04-25T15:19:54.019861+0000","last_clean":"2024-04-25T15:19:54.019861+0000","last_became_active":"2024-04-25T15:19:45.466054+0000","last_became_peered":"2024-04-25T15:19:45.466054+0000","last_unstale":"2024-04-25T15:19:54.019861+0000","last_undegraded":"2024-04-25T15:19:54.019861+0000","last_fullsized":"2024-04-25T15:19:54.019861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:44.430319+0000","last_clean_scrub_stamp":"2024-04-25T15:19:44.430319+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:33.631245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491098+0000","last_change":"2024-04-25T15:19:46.471011+0000","last_active":"2024-04-25T15:19:52.491098+0000","last_peered":"2024-04-25T15:19:52.491098+0000","last_clean":"2024-04-25T15:19:52.491098+0000","last_became_active":"2024-04-25T15:19:46.470620+0000","last_became_peered":"2024-04-25T15:19:46.470620+0000","last_unstale":"2024-04-25T15:19:52.491098+0000","last_undegraded":"2024-04-25T15:19:52.491098+0000","last_fullsized":"2024-04-25T15:19:52.491098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:15.434932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021206+0000","last_change":"2024-04-25T15:19:50.517337+0000","last_active":"2024-04-25T15:19:54.021206+0000","last_peered":"2024-04-25T15:19:54.021206+0000","last_clean":"2024-04-25T15:19:54.021206+0000","last_became_active":"2024-04-25T15:19:50.516848+0000","last_became_peered":"2024-04-25T15:19:50.516848+0000","last_unstale":"2024-04-25T15:19:54.021206+0000","last_undegraded":"2024-04-25T15:19:54.021206+0000","last_fullsized":"2024-04-25T15:19:54.021206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:49.473435+0000","last_clean_scrub_stamp":"2024-04-25T15:19:49.473435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:21:22.692161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018022+0000","last_change":"2024-04-25T15:19:46.476012+0000","last_active":"2024-04-25T15:19:54.018022+0000","last_peered":"2024-04-25T15:19:54.018022+0000","last_clean":"2024-04-25T15:19:54.018022+0000","last_became_active":"2024-04-25T15:19:46.475700+0000","last_became_peered":"2024-04-25T15:19:46.475700+0000","last_unstale":"2024-04-25T15:19:54.018022+0000","last_undegraded":"2024-04-25T15:19:54.018022+0000","last_fullsized":"2024-04-25T15:19:54.018022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020188+0000","last_change":"2024-04-25T15:19:46.484292+0000","last_active":"2024-04-25T15:19:54.020188+0000","last_peered":"2024-04-25T15:19:54.020188+0000","last_clean":"2024-04-25T15:19:54.020188+0000","last_became_active":"2024-04-25T15:19:46.483963+0000","last_became_peered":"2024-04-25T15:19:46.483963+0000","last_unstale":"2024-04-25T15:19:54.020188+0000","last_undegraded":"2024-04-25T15:19:54.020188+0000","last_fullsized":"2024-04-25T15:19:54.020188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490912+0000","last_change":"2024-04-25T15:19:46.484902+0000","last_active":"2024-04-25T15:19:52.490912+0000","last_peered":"2024-04-25T15:19:52.490912+0000","last_clean":"2024-04-25T15:19:52.490912+0000","last_became_active":"2024-04-25T15:19:46.482984+0000","last_became_peered":"2024-04-25T15:19:46.482984+0000","last_unstale":"2024-04-25T15:19:52.490912+0000","last_undegraded":"2024-04-25T15:19:52.490912+0000","last_fullsized":"2024-04-25T15:19:52.490912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:42.718513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488966+0000","last_change":"2024-04-25T15:19:46.468583+0000","last_active":"2024-04-25T15:19:52.488966+0000","last_peered":"2024-04-25T15:19:52.488966+0000","last_clean":"2024-04-25T15:19:52.488966+0000","last_became_active":"2024-04-25T15:19:46.468260+0000","last_became_peered":"2024-04-25T15:19:46.468260+0000","last_unstale":"2024-04-25T15:19:52.488966+0000","last_undegraded":"2024-04-25T15:19:52.488966+0000","last_fullsized":"2024-04-25T15:19:52.488966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.021455+0000","last_change":"2024-04-25T15:19:46.478008+0000","last_active":"2024-04-25T15:19:54.021455+0000","last_peered":"2024-04-25T15:19:54.021455+0000","last_clean":"2024-04-25T15:19:54.021455+0000","last_became_active":"2024-04-25T15:19:46.477687+0000","last_became_peered":"2024-04-25T15:19:46.477687+0000","last_unstale":"2024-04-25T15:19:54.021455+0000","last_undegraded":"2024-04-25T15:19:54.021455+0000","last_fullsized":"2024-04-25T15:19:54.021455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809819+0000","last_change":"2024-04-25T15:19:46.475994+0000","last_active":"2024-04-25T15:19:53.809819+0000","last_peered":"2024-04-25T15:19:53.809819+0000","last_clean":"2024-04-25T15:19:53.809819+0000","last_became_active":"2024-04-25T15:19:46.475566+0000","last_became_peered":"2024-04-25T15:19:46.475566+0000","last_unstale":"2024-04-25T15:19:53.809819+0000","last_undegraded":"2024-04-25T15:19:53.809819+0000","last_fullsized":"2024-04-25T15:19:53.809819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018933+0000","last_change":"2024-04-25T15:19:46.480931+0000","last_active":"2024-04-25T15:19:54.018933+0000","last_peered":"2024-04-25T15:19:54.018933+0000","last_clean":"2024-04-25T15:19:54.018933+0000","last_became_active":"2024-04-25T15:19:46.480607+0000","last_became_peered":"2024-04-25T15:19:46.480607+0000","last_unstale":"2024-04-25T15:19:54.018933+0000","last_undegraded":"2024-04-25T15:19:54.018933+0000","last_fullsized":"2024-04-25T15:19:54.018933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020473+0000","last_change":"2024-04-25T15:19:46.478746+0000","last_active":"2024-04-25T15:19:54.020473+0000","last_peered":"2024-04-25T15:19:54.020473+0000","last_clean":"2024-04-25T15:19:54.020473+0000","last_became_active":"2024-04-25T15:19:46.478389+0000","last_became_peered":"2024-04-25T15:19:46.478389+0000","last_unstale":"2024-04-25T15:19:54.020473+0000","last_undegraded":"2024-04-25T15:19:54.020473+0000","last_fullsized":"2024-04-25T15:19:54.020473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490185+0000","last_change":"2024-04-25T15:19:46.482464+0000","last_active":"2024-04-25T15:19:52.490185+0000","last_peered":"2024-04-25T15:19:52.490185+0000","last_clean":"2024-04-25T15:19:52.490185+0000","last_became_active":"2024-04-25T15:19:46.482115+0000","last_became_peered":"2024-04-25T15:19:46.482115+0000","last_unstale":"2024-04-25T15:19:52.490185+0000","last_undegraded":"2024-04-25T15:19:52.490185+0000","last_fullsized":"2024-04-25T15:19:52.490185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020755+0000","last_change":"2024-04-25T15:19:46.477554+0000","last_active":"2024-04-25T15:19:54.020755+0000","last_peered":"2024-04-25T15:19:54.020755+0000","last_clean":"2024-04-25T15:19:54.020755+0000","last_became_active":"2024-04-25T15:19:46.477240+0000","last_became_peered":"2024-04-25T15:19:46.477240+0000","last_unstale":"2024-04-25T15:19:54.020755+0000","last_undegraded":"2024-04-25T15:19:54.020755+0000","last_fullsized":"2024-04-25T15:19:54.020755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490811+0000","last_change":"2024-04-25T15:19:46.486093+0000","last_active":"2024-04-25T15:19:52.490811+0000","last_peered":"2024-04-25T15:19:52.490811+0000","last_clean":"2024-04-25T15:19:52.490811+0000","last_became_active":"2024-04-25T15:19:46.484171+0000","last_became_peered":"2024-04-25T15:19:46.484171+0000","last_unstale":"2024-04-25T15:19:52.490811+0000","last_undegraded":"2024-04-25T15:19:52.490811+0000","last_fullsized":"2024-04-25T15:19:52.490811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:57.317332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.488875+0000","last_change":"2024-04-25T15:19:46.472207+0000","last_active":"2024-04-25T15:19:52.488875+0000","last_peered":"2024-04-25T15:19:52.488875+0000","last_clean":"2024-04-25T15:19:52.488875+0000","last_became_active":"2024-04-25T15:19:46.471780+0000","last_became_peered":"2024-04-25T15:19:46.471780+0000","last_unstale":"2024-04-25T15:19:52.488875+0000","last_undegraded":"2024-04-25T15:19:52.488875+0000","last_fullsized":"2024-04-25T15:19:52.488875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018502+0000","last_change":"2024-04-25T15:19:46.477605+0000","last_active":"2024-04-25T15:19:54.018502+0000","last_peered":"2024-04-25T15:19:54.018502+0000","last_clean":"2024-04-25T15:19:54.018502+0000","last_became_active":"2024-04-25T15:19:46.477324+0000","last_became_peered":"2024-04-25T15:19:46.477324+0000","last_unstale":"2024-04-25T15:19:54.018502+0000","last_undegraded":"2024-04-25T15:19:54.018502+0000","last_fullsized":"2024-04-25T15:19:54.018502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397007+0000","last_change":"2024-04-25T15:19:46.468508+0000","last_active":"2024-04-25T15:19:53.397007+0000","last_peered":"2024-04-25T15:19:53.397007+0000","last_clean":"2024-04-25T15:19:53.397007+0000","last_became_active":"2024-04-25T15:19:46.468170+0000","last_became_peered":"2024-04-25T15:19:46.468170+0000","last_unstale":"2024-04-25T15:19:53.397007+0000","last_undegraded":"2024-04-25T15:19:53.397007+0000","last_fullsized":"2024-04-25T15:19:53.397007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:00.962714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020658+0000","last_change":"2024-04-25T15:19:46.475011+0000","last_active":"2024-04-25T15:19:54.020658+0000","last_peered":"2024-04-25T15:19:54.020658+0000","last_clean":"2024-04-25T15:19:54.020658+0000","last_became_active":"2024-04-25T15:19:46.474637+0000","last_became_peered":"2024-04-25T15:19:46.474637+0000","last_unstale":"2024-04-25T15:19:54.020658+0000","last_undegraded":"2024-04-25T15:19:54.020658+0000","last_fullsized":"2024-04-25T15:19:54.020658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:24.971213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491965+0000","last_change":"2024-04-25T15:19:46.483654+0000","last_active":"2024-04-25T15:19:52.491965+0000","last_peered":"2024-04-25T15:19:52.491965+0000","last_clean":"2024-04-25T15:19:52.491965+0000","last_became_active":"2024-04-25T15:19:46.483291+0000","last_became_peered":"2024-04-25T15:19:46.483291+0000","last_unstale":"2024-04-25T15:19:52.491965+0000","last_undegraded":"2024-04-25T15:19:52.491965+0000","last_fullsized":"2024-04-25T15:19:52.491965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:58.528155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490226+0000","last_change":"2024-04-25T15:19:46.483513+0000","last_active":"2024-04-25T15:19:52.490226+0000","last_peered":"2024-04-25T15:19:52.490226+0000","last_clean":"2024-04-25T15:19:52.490226+0000","last_became_active":"2024-04-25T15:19:46.483158+0000","last_became_peered":"2024-04-25T15:19:46.483158+0000","last_unstale":"2024-04-25T15:19:52.490226+0000","last_undegraded":"2024-04-25T15:19:52.490226+0000","last_fullsized":"2024-04-25T15:19:52.490226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809438+0000","last_change":"2024-04-25T15:19:46.479148+0000","last_active":"2024-04-25T15:19:53.809438+0000","last_peered":"2024-04-25T15:19:53.809438+0000","last_clean":"2024-04-25T15:19:53.809438+0000","last_became_active":"2024-04-25T15:19:46.478829+0000","last_became_peered":"2024-04-25T15:19:46.478829+0000","last_unstale":"2024-04-25T15:19:53.809438+0000","last_undegraded":"2024-04-25T15:19:53.809438+0000","last_fullsized":"2024-04-25T15:19:53.809438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:20.532556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489010+0000","last_change":"2024-04-25T15:19:46.471858+0000","last_active":"2024-04-25T15:19:52.489010+0000","last_peered":"2024-04-25T15:19:52.489010+0000","last_clean":"2024-04-25T15:19:52.489010+0000","last_became_active":"2024-04-25T15:19:46.471517+0000","last_became_peered":"2024-04-25T15:19:46.471517+0000","last_unstale":"2024-04-25T15:19:52.489010+0000","last_undegraded":"2024-04-25T15:19:52.489010+0000","last_fullsized":"2024-04-25T15:19:52.489010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:58.617984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020190+0000","last_change":"2024-04-25T15:19:46.473809+0000","last_active":"2024-04-25T15:19:54.020190+0000","last_peered":"2024-04-25T15:19:54.020190+0000","last_clean":"2024-04-25T15:19:54.020190+0000","last_became_active":"2024-04-25T15:19:46.473479+0000","last_became_peered":"2024-04-25T15:19:46.473479+0000","last_unstale":"2024-04-25T15:19:54.020190+0000","last_undegraded":"2024-04-25T15:19:54.020190+0000","last_fullsized":"2024-04-25T15:19:54.020190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:34.061516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490638+0000","last_change":"2024-04-25T15:19:46.473643+0000","last_active":"2024-04-25T15:19:52.490638+0000","last_peered":"2024-04-25T15:19:52.490638+0000","last_clean":"2024-04-25T15:19:52.490638+0000","last_became_active":"2024-04-25T15:19:46.473190+0000","last_became_peered":"2024-04-25T15:19:46.473190+0000","last_unstale":"2024-04-25T15:19:52.490638+0000","last_undegraded":"2024-04-25T15:19:52.490638+0000","last_fullsized":"2024-04-25T15:19:52.490638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:30.309739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018544+0000","last_change":"2024-04-25T15:19:46.488270+0000","last_active":"2024-04-25T15:19:54.018544+0000","last_peered":"2024-04-25T15:19:54.018544+0000","last_clean":"2024-04-25T15:19:54.018544+0000","last_became_active":"2024-04-25T15:19:46.487935+0000","last_became_peered":"2024-04-25T15:19:46.487935+0000","last_unstale":"2024-04-25T15:19:54.018544+0000","last_undegraded":"2024-04-25T15:19:54.018544+0000","last_fullsized":"2024-04-25T15:19:54.018544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:03.227361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.018847+0000","last_change":"2024-04-25T15:19:46.488350+0000","last_active":"2024-04-25T15:19:54.018847+0000","last_peered":"2024-04-25T15:19:54.018847+0000","last_clean":"2024-04-25T15:19:54.018847+0000","last_became_active":"2024-04-25T15:19:46.487997+0000","last_became_peered":"2024-04-25T15:19:46.487997+0000","last_unstale":"2024-04-25T15:19:54.018847+0000","last_undegraded":"2024-04-25T15:19:54.018847+0000","last_fullsized":"2024-04-25T15:19:54.018847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397147+0000","last_change":"2024-04-25T15:19:46.486142+0000","last_active":"2024-04-25T15:19:53.397147+0000","last_peered":"2024-04-25T15:19:53.397147+0000","last_clean":"2024-04-25T15:19:53.397147+0000","last_became_active":"2024-04-25T15:19:46.485774+0000","last_became_peered":"2024-04-25T15:19:46.485774+0000","last_unstale":"2024-04-25T15:19:53.397147+0000","last_undegraded":"2024-04-25T15:19:53.397147+0000","last_fullsized":"2024-04-25T15:19:53.397147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:16.297767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:54.020777+0000","last_change":"2024-04-25T15:19:46.486137+0000","last_active":"2024-04-25T15:19:54.020777+0000","last_peered":"2024-04-25T15:19:54.020777+0000","last_clean":"2024-04-25T15:19:54.020777+0000","last_became_active":"2024-04-25T15:19:46.485814+0000","last_became_peered":"2024-04-25T15:19:46.485814+0000","last_unstale":"2024-04-25T15:19:54.020777+0000","last_undegraded":"2024-04-25T15:19:54.020777+0000","last_fullsized":"2024-04-25T15:19:54.020777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:09.799172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.469644+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469320+0000","last_became_peered":"2024-04-25T15:19:46.469320+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.490739+0000","last_change":"2024-04-25T15:19:46.486357+0000","last_active":"2024-04-25T15:19:52.490739+0000","last_peered":"2024-04-25T15:19:52.490739+0000","last_clean":"2024-04-25T15:19:52.490739+0000","last_became_active":"2024-04-25T15:19:46.486016+0000","last_became_peered":"2024-04-25T15:19:46.486016+0000","last_unstale":"2024-04-25T15:19:52.490739+0000","last_undegraded":"2024-04-25T15:19:52.490739+0000","last_fullsized":"2024-04-25T15:19:52.490739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:10.390043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.396418+0000","last_change":"2024-04-25T15:19:46.470290+0000","last_active":"2024-04-25T15:19:53.396418+0000","last_peered":"2024-04-25T15:19:53.396418+0000","last_clean":"2024-04-25T15:19:53.396418+0000","last_became_active":"2024-04-25T15:19:46.469965+0000","last_became_peered":"2024-04-25T15:19:46.469965+0000","last_unstale":"2024-04-25T15:19:53.396418+0000","last_undegraded":"2024-04-25T15:19:53.396418+0000","last_fullsized":"2024-04-25T15:19:53.396418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:09.327229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.491250+0000","last_change":"2024-04-25T15:19:46.483532+0000","last_active":"2024-04-25T15:19:52.491250+0000","last_peered":"2024-04-25T15:19:52.491250+0000","last_clean":"2024-04-25T15:19:52.491250+0000","last_became_active":"2024-04-25T15:19:46.483242+0000","last_became_peered":"2024-04-25T15:19:46.483242+0000","last_unstale":"2024-04-25T15:19:52.491250+0000","last_undegraded":"2024-04-25T15:19:52.491250+0000","last_fullsized":"2024-04-25T15:19:52.491250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:52:27.913274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:52.489031+0000","last_change":"2024-04-25T15:19:46.467386+0000","last_active":"2024-04-25T15:19:52.489031+0000","last_peered":"2024-04-25T15:19:52.489031+0000","last_clean":"2024-04-25T15:19:52.489031+0000","last_became_active":"2024-04-25T15:19:46.467034+0000","last_became_peered":"2024-04-25T15:19:46.467034+0000","last_unstale":"2024-04-25T15:19:52.489031+0000","last_undegraded":"2024-04-25T15:19:52.489031+0000","last_fullsized":"2024-04-25T15:19:52.489031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:23.069717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.809963+0000","last_change":"2024-04-25T15:19:46.472318+0000","last_active":"2024-04-25T15:19:53.809963+0000","last_peered":"2024-04-25T15:19:53.809963+0000","last_clean":"2024-04-25T15:19:53.809963+0000","last_became_active":"2024-04-25T15:19:46.471982+0000","last_became_peered":"2024-04-25T15:19:46.471982+0000","last_unstale":"2024-04-25T15:19:53.809963+0000","last_undegraded":"2024-04-25T15:19:53.809963+0000","last_fullsized":"2024-04-25T15:19:53.809963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:16.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T15:19:53.397045+0000","last_change":"2024-04-25T15:19:46.473366+0000","last_active":"2024-04-25T15:19:53.397045+0000","last_peered":"2024-04-25T15:19:53.397045+0000","last_clean":"2024-04-25T15:19:53.397045+0000","last_became_active":"2024-04-25T15:19:46.472969+0000","last_became_peered":"2024-04-25T15:19:46.472969+0000","last_unstale":"2024-04-25T15:19:53.397045+0000","last_undegraded":"2024-04-25T15:19:53.397045+0000","last_fullsized":"2024-04-25T15:19:53.397045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:45.436281+0000","last_clean_scrub_stamp":"2024-04-25T15:19:45.436281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:59:43.070586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:20:14.470 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:20:14.470 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T15:20:14.470 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T15:20:14.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:20:14.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.613+0000 7f4021892640 1 Processor -- start 2024-04-25T15:20:14.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.613+0000 7f4021892640 1 -- start start 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4021892640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f401c0d0730 0x7f401c0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4021892640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f401c0b9200 con 0x7f401c06ccc0 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4021892640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f401c0d1070 con 0x7f401c0d4f60 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4021892640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f401c06c550 con 0x7f401c0d0730 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f401b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f401c06ccc0 0x7f401c06d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40650/0 (socket says 172.21.15.146:40650) 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f401b7fe640 1 -- 172.21.15.146:0/4207920284 learned_addr learned my addr 172.21.15.146:0/4207920284 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:14.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f401a7fc640 1 --2- 172.21.15.146:0/4207920284 >> v2:172.21.15.177:3300/0 conn(0x7f401c0d0730 0x7f401c0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257678209 0 0) 0x7f400c001110 con 0x7f401c06ccc0 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4008003520 con 0x7f401c06ccc0 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1346657744 0 0) 0x7f4008003520 con 0x7f401c06ccc0 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 >> v2:172.21.15.177:3300/0 conn(0x7f401c0d0730 msgr2=0x7f401c0d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 --2- 172.21.15.146:0/4207920284 >> v2:172.21.15.177:3300/0 conn(0x7f401c0d0730 0x7f401c0d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 >> v1:172.21.15.177:6789/0 conn(0x7f401c0d4f60 legacy=0x7f401c0d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c0d1310 con 0x7f401c06ccc0 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f400c001840 con 0x7f401c06ccc0 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 >> v1:172.21.15.146:6789/0 conn(0x7f401c06ccc0 legacy=0x7f401c06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40080039b0 0x7f4008003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4008004800 0x7f4008004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4008005350 0x7f400800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f401c06c550 con 0x7f4008004800 2024-04-25T15:20:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f400800c280 con 0x7f40080039b0 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.615+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f400800c480 con 0x7f4008005350 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4008004800 0x7f4008004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401affd640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40080039b0 0x7f4008003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401b7fe640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4008005350 0x7f400800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 -- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4008005350 msgr2=0x7f400800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4008005350 0x7f400800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 -- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40080039b0 msgr2=0x7f4008003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40080039b0 0x7f4008003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 -- 172.21.15.146:0/4207920284 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4008003520 con 0x7f4008004800 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401b7fe640 1 -- 172.21.15.146:0/4207920284 reap_dead start 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.616+0000 7f401a7fc640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4008004800 0x7f4008004c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4004009ac0 tx=0x7f4008005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b6216cd1c6ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004036070 con 0x7f4008004800 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4004031400 con 0x7f4008004800 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4019ffb640 1 -- 172.21.15.146:0/4207920284 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004009690 con 0x7f4008004800 2024-04-25T15:20:14.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 -- 172.21.15.146:0/4207920284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4008004800 msgr2=0x7f4008004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4008004800 0x7f4008004c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4004009ac0 tx=0x7f4008005830 comp rx=0 tx=0).stop 2024-04-25T15:20:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 -- 172.21.15.146:0/4207920284 shutdown_connections 2024-04-25T15:20:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 --2- 172.21.15.146:0/4207920284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4008004800 0x7f4008004c10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 -- 172.21.15.146:0/4207920284 >> 172.21.15.146:0/4207920284 conn(0x7f401c0d2450 msgr2=0x7f401c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 -- 172.21.15.146:0/4207920284 shutdown_connections 2024-04-25T15:20:14.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.617+0000 7f4021892640 1 -- 172.21.15.146:0/4207920284 wait complete. 2024-04-25T15:20:14.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.619+0000 7f4021892640 1 Processor -- start 2024-04-25T15:20:14.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.619+0000 7f4021892640 1 -- start start 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.620+0000 7f4021892640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4008004800 0x7f401c0cec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 0x7f401c15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f401c15e6b0 0x7f401c15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f401c0b5150 con 0x7f401c15e2a0 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f401c06c830 con 0x7f4008004800 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f401c0b52d0 con 0x7f401c15e6b0 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 0x7f401c15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 0x7f401c15aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60344/0 (socket says 172.21.15.146:60344) 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 -- 172.21.15.146:0/3642174703 learned_addr learned my addr 172.21.15.146:0/3642174703 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401affd640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4008004800 0x7f401c0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401b7fe640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f401c15e6b0 0x7f401c15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 -- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f401c15e6b0 msgr2=0x7f401c15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f401c15e6b0 0x7f401c15afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 -- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4008004800 msgr2=0x7f401c0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4008004800 0x7f401c0cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 -- 172.21.15.146:0/3642174703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c0cf150 con 0x7f401c15e2a0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f401a7fc640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 0x7f401c15aa60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4004009be0 tx=0x7f4004004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400403f020 con 0x7f401c15e2a0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4004037040 con 0x7f401c15e2a0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004036070 con 0x7f401c15e2a0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401c1e8b50 con 0x7f401c15e2a0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.621+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401c1e9110 con 0x7f401c15e2a0 2024-04-25T15:20:14.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.622+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401c06a0e0 con 0x7f401c15e2a0 2024-04-25T15:20:14.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.623+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4004042020 con 0x7f401c15e2a0 2024-04-25T15:20:14.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.623+0000 7f4013fff640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ffc0744b0 0x7f3ffc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.624+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f40040b5680 con 0x7f401c15e2a0 2024-04-25T15:20:14.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.624+0000 7f401affd640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ffc0744b0 0x7f3ffc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.624+0000 7f401affd640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ffc0744b0 0x7f3ffc0769c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4014000f80 tx=0x7f4014009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.628+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4004049030 con 0x7f401c15e2a0 2024-04-25T15:20:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.840+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f401c0d4100 con 0x7f401c15e2a0 2024-04-25T15:20:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.841+0000 7f4013fff640 1 -- 172.21.15.146:0/3642174703 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f400407f540 con 0x7f401c15e2a0 2024-04-25T15:20:14.842 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:14.843 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T15:20:14.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ffc0744b0 msgr2=0x7f3ffc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ffc0744b0 0x7f3ffc0769c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4014000f80 tx=0x7f4014009040 comp rx=0 tx=0).stop 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 msgr2=0x7f401c15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 0x7f401c15aa60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4004009be0 tx=0x7f4004004730 comp rx=0 tx=0).stop 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 shutdown_connections 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ffc0744b0 0x7f3ffc0769c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f401c15e6b0 0x7f401c15afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f401c15e2a0 0x7f401c15aa60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 --2- 172.21.15.146:0/3642174703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4008004800 0x7f401c0cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.842+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 >> 172.21.15.146:0/3642174703 conn(0x7f401c0d2450 msgr2=0x7f401c068770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.843+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 shutdown_connections 2024-04-25T15:20:14.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.843+0000 7f4021892640 1 -- 172.21.15.146:0/3642174703 wait complete. 2024-04-25T15:20:14.856 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T15:20:14.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:14.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.962+0000 7f8620d27640 1 Processor -- start 2024-04-25T15:20:14.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.962+0000 7f8620d27640 1 -- start start 2024-04-25T15:20:14.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8620d27640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f861c0d4f60 0x7f861c0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8620d27640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d0020 0x7f861c0d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8620d27640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f861c163d80 0x7f861c164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8620d27640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f861c06c550 con 0x7f861c163d80 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8620d27640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f861c06c830 con 0x7f861c0d4f60 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8620d27640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f861c0b9200 con 0x7f861c0d0020 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f861ad9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f861c163d80 0x7f861c164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f861ad9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f861c163d80 0x7f861c164190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60348/0 (socket says 172.21.15.146:60348) 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f861ad9d640 1 -- 172.21.15.146:0/1129494686 learned_addr learned my addr 172.21.15.146:0/1129494686 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f8619d9b640 1 --2- 172.21.15.146:0/1129494686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d0020 0x7f861c0d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.963+0000 7f861a59c640 1 --2- 172.21.15.146:0/1129494686 >> v2:172.21.15.177:3300/0 conn(0x7f861c0d4f60 0x7f861c0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861ad9d640 1 -- 172.21.15.146:0/1129494686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d0020 msgr2=0x7f861c0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861ad9d640 1 --2- 172.21.15.146:0/1129494686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d0020 0x7f861c0d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861ad9d640 1 -- 172.21.15.146:0/1129494686 >> v2:172.21.15.177:3300/0 conn(0x7f861c0d4f60 msgr2=0x7f861c0d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861ad9d640 1 --2- 172.21.15.146:0/1129494686 >> v2:172.21.15.177:3300/0 conn(0x7f861c0d4f60 0x7f861c0d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861ad9d640 1 -- 172.21.15.146:0/1129494686 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c0d0970 con 0x7f861c163d80 2024-04-25T15:20:14.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861ad9d640 1 --2- 172.21.15.146:0/1129494686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c164190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f861c0d5930 tx=0x7f861402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732c3844eb955154 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861959a640 1 -- 172.21.15.146:0/1129494686 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614037070 con 0x7f861c163d80 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861959a640 1 -- 172.21.15.146:0/1129494686 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861402ae00 con 0x7f861c163d80 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.964+0000 7f861959a640 1 -- 172.21.15.146:0/1129494686 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614032660 con 0x7f861c163d80 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 -- 172.21.15.146:0/1129494686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 msgr2=0x7f861c164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 --2- 172.21.15.146:0/1129494686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c164190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f861c0d5930 tx=0x7f861402a0b0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 -- 172.21.15.146:0/1129494686 shutdown_connections 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 --2- 172.21.15.146:0/1129494686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c164190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 --2- 172.21.15.146:0/1129494686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d0020 0x7f861c0d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 --2- 172.21.15.146:0/1129494686 >> v2:172.21.15.177:3300/0 conn(0x7f861c0d4f60 0x7f861c0d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 -- 172.21.15.146:0/1129494686 >> 172.21.15.146:0/1129494686 conn(0x7f861c0d2450 msgr2=0x7f861c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 -- 172.21.15.146:0/1129494686 shutdown_connections 2024-04-25T15:20:14.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.965+0000 7f8620d27640 1 -- 172.21.15.146:0/1129494686 wait complete. 2024-04-25T15:20:14.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.967+0000 7f8620d27640 1 Processor -- start 2024-04-25T15:20:14.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.967+0000 7f8620d27640 1 -- start start 2024-04-25T15:20:14.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.968+0000 7f8620d27640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f861c0d0020 0x7f861c15f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.968+0000 7f8620d27640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d4f60 0x7f861c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.968+0000 7f8620d27640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.968+0000 7f8620d27640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f861c0d15f0 con 0x7f861c163d80 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.968+0000 7f8620d27640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f861c0d1070 con 0x7f861c0d0020 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.968+0000 7f8620d27640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f861c0b5150 con 0x7f861c0d4f60 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f861c0d0020 0x7f861c15f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c0ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60362/0 (socket says 172.21.15.146:60362) 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 -- 172.21.15.146:0/809606850 learned_addr learned my addr 172.21.15.146:0/809606850 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f8619d9b640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d4f60 0x7f861c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 -- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d4f60 msgr2=0x7f861c15def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d4f60 0x7f861c15def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 -- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f861c0d0020 msgr2=0x7f861c15f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:14.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f861c0d0020 0x7f861c15f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 -- 172.21.15.146:0/809606850 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c0cecb0 con 0x7f861c163d80 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f861ad9d640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c0ce770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f861402a0f0 tx=0x7f86140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614041030 con 0x7f861c163d80 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861402ac00 con 0x7f861c163d80 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861c15e8d0 con 0x7f861c163d80 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614037040 con 0x7f861c163d80 2024-04-25T15:20:14.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.969+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c15ee90 con 0x7f861c163d80 2024-04-25T15:20:14.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.970+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861c15e640 con 0x7f861c163d80 2024-04-25T15:20:14.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.971+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8614044070 con 0x7f861c163d80 2024-04-25T15:20:14.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.971+0000 7f8612ffd640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f85fc0744e0 0x7f85fc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:14.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.972+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f86140b5780 con 0x7f861c163d80 2024-04-25T15:20:14.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.972+0000 7f861a59c640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f85fc0744e0 0x7f85fc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:14.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.972+0000 7f861a59c640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f85fc0744e0 0x7f85fc0769f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f861c0b4880 tx=0x7f860c009830 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:14.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:14.976+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f861404b070 con 0x7f861c163d80 2024-04-25T15:20:15.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.153+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f861c06a0e0 con 0x7f861c163d80 2024-04-25T15:20:15.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.155+0000 7f8612ffd640 1 -- 172.21.15.146:0/809606850 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7f861c0d0970 con 0x7f861c163d80 2024-04-25T15:20:15.157 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:15.157 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f85fc0744e0 msgr2=0x7f85fc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f85fc0744e0 0x7f85fc0769f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f861c0b4880 tx=0x7f860c009830 comp rx=0 tx=0).stop 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 msgr2=0x7f861c0ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c0ce770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f861402a0f0 tx=0x7f86140046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 shutdown_connections 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f85fc0744e0 0x7f85fc0769f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861c163d80 0x7f861c0ce770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f861c0d4f60 0x7f861c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 --2- 172.21.15.146:0/809606850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f861c0d0020 0x7f861c15f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 >> 172.21.15.146:0/809606850 conn(0x7f861c0d2450 msgr2=0x7f861c0b5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:15.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 shutdown_connections 2024-04-25T15:20:15.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.157+0000 7f8620d27640 1 -- 172.21.15.146:0/809606850 wait complete. 2024-04-25T15:20:15.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:15.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:15.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.280+0000 7f2f70070640 1 Processor -- start 2024-04-25T15:20:15.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.280+0000 7f2f70070640 1 -- start start 2024-04-25T15:20:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f70070640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f680bb450 0x7f2f680d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f70070640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2f680d0de0 0x7f2f680b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f70070640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f680b5230 con 0x7f2f68163c00 2024-04-25T15:20:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f70070640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f2f6806c8c0 con 0x7f2f680bb450 2024-04-25T15:20:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f70070640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2f6806cba0 con 0x7f2f680d0de0 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f6d60b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f680bb450 0x7f2f680d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f6d60b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f680bb450 0x7f2f680d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60374/0 (socket says 172.21.15.146:60374) 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f6d60b640 1 -- 172.21.15.146:0/4245814163 learned_addr learned my addr 172.21.15.146:0/4245814163 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.281+0000 7f2f6de0c640 1 --1- 172.21.15.146:0/4245814163 >> v1:172.21.15.146:6789/0 conn(0x7f2f68163c00 0x7f2f6806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40662/0 (socket says 172.21.15.146:40662) 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6e60d640 1 --2- 172.21.15.146:0/4245814163 >> v2:172.21.15.177:3300/0 conn(0x7f2f680d0de0 0x7f2f680b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6d60b640 1 -- 172.21.15.146:0/4245814163 >> v2:172.21.15.177:3300/0 conn(0x7f2f680d0de0 msgr2=0x7f2f680b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6d60b640 1 --2- 172.21.15.146:0/4245814163 >> v2:172.21.15.177:3300/0 conn(0x7f2f680d0de0 0x7f2f680b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6d60b640 1 -- 172.21.15.146:0/4245814163 >> v1:172.21.15.146:6789/0 conn(0x7f2f68163c00 legacy=0x7f2f6806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6ce0a640 1 -- 172.21.15.146:0/4245814163 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173944721 0 0) 0x7f2f680b5230 con 0x7f2f68163c00 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6d60b640 1 -- 172.21.15.146:0/4245814163 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6806ce80 con 0x7f2f680bb450 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.282+0000 7f2f6e60d640 1 --2- 172.21.15.146:0/4245814163 >> v2:172.21.15.177:3300/0 conn(0x7f2f680d0de0 0x7f2f680b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f6d60b640 1 --2- 172.21.15.146:0/4245814163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f680d08a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009870 tx=0x7f2f5802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31acca27c7659dd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f6ce0a640 1 -- 172.21.15.146:0/4245814163 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58037070 con 0x7f2f680bb450 2024-04-25T15:20:15.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f6ce0a640 1 -- 172.21.15.146:0/4245814163 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f58032440 con 0x7f2f680bb450 2024-04-25T15:20:15.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f6ce0a640 1 -- 172.21.15.146:0/4245814163 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58032810 con 0x7f2f680bb450 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 -- 172.21.15.146:0/4245814163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 msgr2=0x7f2f680d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 --2- 172.21.15.146:0/4245814163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f680d08a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009870 tx=0x7f2f5802a210 comp rx=0 tx=0).stop 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 -- 172.21.15.146:0/4245814163 shutdown_connections 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 --2- 172.21.15.146:0/4245814163 >> v2:172.21.15.177:3300/0 conn(0x7f2f680d0de0 0x7f2f680b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 --2- 172.21.15.146:0/4245814163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f680d08a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 -- 172.21.15.146:0/4245814163 >> 172.21.15.146:0/4245814163 conn(0x7f2f680b7970 msgr2=0x7f2f680b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 -- 172.21.15.146:0/4245814163 shutdown_connections 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.283+0000 7f2f70070640 1 -- 172.21.15.146:0/4245814163 wait complete. 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.285+0000 7f2f70070640 1 Processor -- start 2024-04-25T15:20:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.285+0000 7f2f70070640 1 -- start start 2024-04-25T15:20:15.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.286+0000 7f2f70070640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f6814c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.286+0000 7f2f70070640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f680d0de0 0x7f2f680ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.286+0000 7f2f70070640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68163c00 0x7f2f680b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.286+0000 7f2f70070640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2f680bb860 con 0x7f2f680bb450 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.286+0000 7f2f70070640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6806c550 con 0x7f2f68163c00 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.286+0000 7f2f70070640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2f680bb9e0 con 0x7f2f680d0de0 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f6814c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f6814c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60390/0 (socket says 172.21.15.146:60390) 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 -- 172.21.15.146:0/3411642501 learned_addr learned my addr 172.21.15.146:0/3411642501 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6e60d640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68163c00 0x7f2f680b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6d60b640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f680d0de0 0x7f2f680ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 -- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f680d0de0 msgr2=0x7f2f680ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f680d0de0 0x7f2f680ceac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 -- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68163c00 msgr2=0x7f2f680b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68163c00 0x7f2f680b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 -- 172.21.15.146:0/3411642501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6814f190 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f6de0c640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f6814c580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c460 tx=0x7f2f6400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64018070 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f64007500 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64004a20 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6814f470 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.287+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f6814fa30 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.288+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f6806a0e0 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.289+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2f64004bc0 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.289+0000 7f2f5e7fc640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2f4c0744b0 0x7f2f4c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.289+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2f64094800 con 0x7f2f680bb450 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.289+0000 7f2f6d60b640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2f4c0744b0 0x7f2f4c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.290+0000 7f2f6d60b640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2f4c0744b0 0x7f2f4c0769c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f58034000 tx=0x7f2f5802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.294+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f6405e600 con 0x7f2f680bb450 2024-04-25T15:20:15.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.468+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f680d3ed0 con 0x7f2f680bb450 2024-04-25T15:20:15.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.468+0000 7f2f5e7fc640 1 -- 172.21.15.146:0/3411642501 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f2f64062900 con 0x7f2f680bb450 2024-04-25T15:20:15.470 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:15.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2f4c0744b0 msgr2=0x7f2f4c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2f4c0744b0 0x7f2f4c0769c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f58034000 tx=0x7f2f5802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 msgr2=0x7f2f6814c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f6814c580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400c460 tx=0x7f2f6400c920 comp rx=0 tx=0).stop 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 shutdown_connections 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2f4c0744b0 0x7f2f4c0769c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68163c00 0x7f2f680b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f680d0de0 0x7f2f680ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 --2- 172.21.15.146:0/3411642501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f680bb450 0x7f2f6814c580 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 >> 172.21.15.146:0/3411642501 conn(0x7f2f680b7970 msgr2=0x7f2f6815fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:15.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 shutdown_connections 2024-04-25T15:20:15.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.470+0000 7f2f70070640 1 -- 172.21.15.146:0/3411642501 wait complete. 2024-04-25T15:20:15.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:15.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.591+0000 7f4620ed9640 1 Processor -- start 2024-04-25T15:20:15.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.591+0000 7f4620ed9640 1 -- start start 2024-04-25T15:20:15.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f4620ed9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f461c08dd50 0x7f461c0b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f4620ed9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c08d590 con 0x7f461c0b8f30 2024-04-25T15:20:15.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f4620ed9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c0d27d0 con 0x7f461c0ce6f0 2024-04-25T15:20:15.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f4620ed9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f461c0b9340 con 0x7f461c08dd50 2024-04-25T15:20:15.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f4613fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f461c0b8f30 0x7f461c0d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40676/0 (socket says 172.21.15.146:40676) 2024-04-25T15:20:15.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f4613fff640 1 -- 172.21.15.146:0/966215260 learned_addr learned my addr 172.21.15.146:0/966215260 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:15.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f461a59c640 1 --2- 172.21.15.146:0/966215260 >> v2:172.21.15.177:3300/0 conn(0x7f461c08dd50 0x7f461c0b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f461a59c640 1 -- 172.21.15.146:0/966215260 >> v1:172.21.15.177:6789/0 conn(0x7f461c0ce6f0 legacy=0x7f461c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f461a59c640 1 -- 172.21.15.146:0/966215260 >> v1:172.21.15.146:6789/0 conn(0x7f461c0b8f30 legacy=0x7f461c0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.593+0000 7f461a59c640 1 -- 172.21.15.146:0/966215260 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c0b48b0 con 0x7f461c08dd50 2024-04-25T15:20:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.594+0000 7f461a59c640 1 --2- 172.21.15.146:0/966215260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c08dd50 0x7f461c0b4ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f460c0097a0 tx=0x7f460c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd9c93d8eeed9769 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.594+0000 7f4619d9b640 1 -- 172.21.15.146:0/966215260 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c037070 con 0x7f461c08dd50 2024-04-25T15:20:15.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4619d9b640 1 -- 172.21.15.146:0/966215260 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460c032440 con 0x7f461c08dd50 2024-04-25T15:20:15.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4619d9b640 1 -- 172.21.15.146:0/966215260 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c032810 con 0x7f461c08dd50 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 -- 172.21.15.146:0/966215260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c08dd50 msgr2=0x7f461c0b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 --2- 172.21.15.146:0/966215260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c08dd50 0x7f461c0b4ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f460c0097a0 tx=0x7f460c02a230 comp rx=0 tx=0).stop 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 -- 172.21.15.146:0/966215260 shutdown_connections 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 --2- 172.21.15.146:0/966215260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c08dd50 0x7f461c0b4ca0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 -- 172.21.15.146:0/966215260 >> 172.21.15.146:0/966215260 conn(0x7f461c0b5430 msgr2=0x7f461c0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 -- 172.21.15.146:0/966215260 shutdown_connections 2024-04-25T15:20:15.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.595+0000 7f4620ed9640 1 -- 172.21.15.146:0/966215260 wait complete. 2024-04-25T15:20:15.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.597+0000 7f4620ed9640 1 Processor -- start 2024-04-25T15:20:15.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.597+0000 7f4620ed9640 1 -- start start 2024-04-25T15:20:15.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 0x7f461c1d3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c0ce6f0 0x7f461c1d5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f461c1d6640 0x7f461c1d6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f461c0d2340 con 0x7f461c0b8f30 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f461c08d8c0 con 0x7f461c0ce6f0 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f461c0d24c0 con 0x7f461c1d6640 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 0x7f461c1d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 0x7f461c1d3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60404/0 (socket says 172.21.15.146:60404) 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 -- 172.21.15.146:0/2761673809 learned_addr learned my addr 172.21.15.146:0/2761673809 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461ad9d640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f461c1d6640 0x7f461c1d6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4613fff640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c0ce6f0 0x7f461c1d5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 -- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f461c1d6640 msgr2=0x7f461c1d6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f461c1d6640 0x7f461c1d6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 -- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c0ce6f0 msgr2=0x7f461c1d5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c0ce6f0 0x7f461c1d5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 -- 172.21.15.146:0/2761673809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c1d41f0 con 0x7f461c0b8f30 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f461a59c640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 0x7f461c1d3930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f460c035040 tx=0x7f460c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c037040 con 0x7f461c0b8f30 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460c030770 con 0x7f461c0b8f30 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c1d44d0 con 0x7f461c0b8f30 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.599+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c030b40 con 0x7f461c0b8f30 2024-04-25T15:20:15.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.600+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c1d4a90 con 0x7f461c0b8f30 2024-04-25T15:20:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.600+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461c06a0e0 con 0x7f461c0b8f30 2024-04-25T15:20:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.601+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f460c044020 con 0x7f461c0b8f30 2024-04-25T15:20:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.601+0000 7f4612ffd640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f45fc0744b0 0x7f45fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.602+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f460c0b5a40 con 0x7f461c0b8f30 2024-04-25T15:20:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.602+0000 7f4613fff640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f45fc0744b0 0x7f45fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.602+0000 7f4613fff640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f45fc0744b0 0x7f45fc0769c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4604000f60 tx=0x7f4604009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.606+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f460c032e20 con 0x7f461c0b8f30 2024-04-25T15:20:15.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.782+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f461c0755d0 con 0x7f461c0b8f30 2024-04-25T15:20:15.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.782+0000 7f4612ffd640 1 -- 172.21.15.146:0/2761673809 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f460c0b5e20 con 0x7f461c0b8f30 2024-04-25T15:20:15.784 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f45fc0744b0 msgr2=0x7f45fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f45fc0744b0 0x7f45fc0769c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4604000f60 tx=0x7f4604009040 comp rx=0 tx=0).stop 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 msgr2=0x7f461c1d3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 0x7f461c1d3930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f460c035040 tx=0x7f460c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 shutdown_connections 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f45fc0744b0 0x7f45fc0769c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f461c1d6640 0x7f461c1d6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f461c0ce6f0 0x7f461c1d5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 --2- 172.21.15.146:0/2761673809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f461c0b8f30 0x7f461c1d3930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 >> 172.21.15.146:0/2761673809 conn(0x7f461c0b5430 msgr2=0x7f461c0ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:15.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 shutdown_connections 2024-04-25T15:20:15.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.784+0000 7f4620ed9640 1 -- 172.21.15.146:0/2761673809 wait complete. 2024-04-25T15:20:15.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:15.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:20:15.845 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:20:15.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.909+0000 7f9995f64640 1 Processor -- start 2024-04-25T15:20:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.909+0000 7f9995f64640 1 -- start start 2024-04-25T15:20:15.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f9995f64640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900b2cd0 0x7f99900d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f9995f64640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9990071d30 0x7f99900cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f9995f64640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99900d2260 con 0x7f99900d3300 2024-04-25T15:20:15.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f9995f64640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f99900d2540 con 0x7f9990071d30 2024-04-25T15:20:15.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f9995f64640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f99900d2820 con 0x7f99900b2cd0 2024-04-25T15:20:15.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9990071d30 0x7f99900cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9990071d30 0x7f99900cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60408/0 (socket says 172.21.15.146:60408) 2024-04-25T15:20:15.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 -- 172.21.15.146:0/1152805320 learned_addr learned my addr 172.21.15.146:0/1152805320 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:15.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998f7fe640 1 --2- 172.21.15.146:0/1152805320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900b2cd0 0x7f99900d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 -- 172.21.15.146:0/1152805320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900b2cd0 msgr2=0x7f99900d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:15.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 --2- 172.21.15.146:0/1152805320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900b2cd0 0x7f99900d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 -- 172.21.15.146:0/1152805320 >> v1:172.21.15.177:6789/0 conn(0x7f99900d3300 legacy=0x7f9990071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.911+0000 7f998ffff640 1 -- 172.21.15.146:0/1152805320 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99900d2b00 con 0x7f9990071d30 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.912+0000 7f998ffff640 1 --2- 172.21.15.146:0/1152805320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9990071d30 0x7f99900cde40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9988009e50 tx=0x7f998802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964e156367867efb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.912+0000 7f998e7fc640 1 -- 172.21.15.146:0/1152805320 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988036070 con 0x7f9990071d30 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.912+0000 7f998e7fc640 1 -- 172.21.15.146:0/1152805320 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9988031600 con 0x7f9990071d30 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.912+0000 7f998e7fc640 1 -- 172.21.15.146:0/1152805320 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99880319d0 con 0x7f9990071d30 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 -- 172.21.15.146:0/1152805320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9990071d30 msgr2=0x7f99900cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 --2- 172.21.15.146:0/1152805320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9990071d30 0x7f99900cde40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9988009e50 tx=0x7f998802a680 comp rx=0 tx=0).stop 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 -- 172.21.15.146:0/1152805320 shutdown_connections 2024-04-25T15:20:15.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 --2- 172.21.15.146:0/1152805320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9990071d30 0x7f99900cde40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 --2- 172.21.15.146:0/1152805320 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900b2cd0 0x7f99900d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 -- 172.21.15.146:0/1152805320 >> 172.21.15.146:0/1152805320 conn(0x7f99900b52e0 msgr2=0x7f99900b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:15.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 -- 172.21.15.146:0/1152805320 shutdown_connections 2024-04-25T15:20:15.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.913+0000 7f9995f64640 1 -- 172.21.15.146:0/1152805320 wait complete. 2024-04-25T15:20:15.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.915+0000 7f9995f64640 1 Processor -- start 2024-04-25T15:20:15.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.915+0000 7f9995f64640 1 -- start start 2024-04-25T15:20:15.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f9995f64640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900b2cd0 0x7f9990157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f9995f64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 0x7f99901e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f9995f64640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900cbae0 0x7f99901ea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f9995f64640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f999008dbf0 con 0x7f99900d3300 2024-04-25T15:20:15.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f9995f64640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99900cfb70 con 0x7f99900b2cd0 2024-04-25T15:20:15.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f9995f64640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f999008dd70 con 0x7f99900cbae0 2024-04-25T15:20:15.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 0x7f99901e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 0x7f99901e9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60422/0 (socket says 172.21.15.146:60422) 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 -- 172.21.15.146:0/1483256572 learned_addr learned my addr 172.21.15.146:0/1483256572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998f7fe640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900b2cd0 0x7f9990157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998ffff640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900cbae0 0x7f99901ea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 -- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900cbae0 msgr2=0x7f99901ea4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900cbae0 0x7f99901ea4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 -- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900b2cd0 msgr2=0x7f9990157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900b2cd0 0x7f9990157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.916+0000 7f998effd640 1 -- 172.21.15.146:0/1483256572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99900cc2c0 con 0x7f99900d3300 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.917+0000 7f998effd640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 0x7f99901e9f90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f998400c3c0 tx=0x7f998400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.917+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99840052d0 con 0x7f99900d3300 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.917+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9984005470 con 0x7f99900d3300 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.917+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99900cc5a0 con 0x7f99900d3300 2024-04-25T15:20:15.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.917+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9984007240 con 0x7f99900d3300 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.917+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99900ccb60 con 0x7f99900d3300 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.918+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f999006a0e0 con 0x7f99900d3300 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.918+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f99840073e0 con 0x7f99900d3300 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.919+0000 7f9977fff640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f997c0744b0 0x7f997c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.919+0000 7f998f7fe640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f997c0744b0 0x7f997c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.919+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9984094970 con 0x7f99900d3300 2024-04-25T15:20:15.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.919+0000 7f998f7fe640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f997c0744b0 0x7f997c0769c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99900b9490 tx=0x7f9980009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:15.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:15.923+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f998405e770 con 0x7f99900d3300 2024-04-25T15:20:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.102+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f99900cc050 con 0x7f99900d3300 2024-04-25T15:20:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.103+0000 7f9977fff640 1 -- 172.21.15.146:0/1483256572 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f9984062a70 con 0x7f99900d3300 2024-04-25T15:20:16.105 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f997c0744b0 msgr2=0x7f997c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f997c0744b0 0x7f997c0769c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99900b9490 tx=0x7f9980009110 comp rx=0 tx=0).stop 2024-04-25T15:20:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 msgr2=0x7f99901e9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 0x7f99901e9f90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f998400c3c0 tx=0x7f998400c7c0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 shutdown_connections 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f997c0744b0 0x7f997c0769c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99900cbae0 0x7f99901ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99900d3300 0x7f99901e9f90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 --2- 172.21.15.146:0/1483256572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99900b2cd0 0x7f9990157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 >> 172.21.15.146:0/1483256572 conn(0x7f99900b52e0 msgr2=0x7f999008c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 shutdown_connections 2024-04-25T15:20:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.105+0000 7f9995f64640 1 -- 172.21.15.146:0/1483256572 wait complete. 2024-04-25T15:20:16.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.268+0000 7f61e14f9640 1 Processor -- start 2024-04-25T15:20:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.268+0000 7f61e14f9640 1 -- start start 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.269+0000 7f61e14f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f61dc163c00 0x7f61dc06d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.269+0000 7f61e14f9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61dc0bb450 0x7f61dc0d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61e14f9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61dc06dea0 con 0x7f61dc0d0de0 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61e14f9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f61dc0b5230 con 0x7f61dc0bb450 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61e14f9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f61dc06c8c0 con 0x7f61dc163c00 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61da7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61dc0bb450 0x7f61dc0d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61db7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f61dc0d0de0 0x7f61dc0b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40684/0 (socket says 172.21.15.146:40684) 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61db7fe640 1 -- 172.21.15.146:0/1243346321 learned_addr learned my addr 172.21.15.146:0/1243346321 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61daffd640 1 --2- 172.21.15.146:0/1243346321 >> v2:172.21.15.177:3300/0 conn(0x7f61dc163c00 0x7f61dc06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61da7fc640 1 -- 172.21.15.146:0/1243346321 >> v2:172.21.15.177:3300/0 conn(0x7f61dc163c00 msgr2=0x7f61dc06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61da7fc640 1 --2- 172.21.15.146:0/1243346321 >> v2:172.21.15.177:3300/0 conn(0x7f61dc163c00 0x7f61dc06d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61da7fc640 1 -- 172.21.15.146:0/1243346321 >> v1:172.21.15.146:6789/0 conn(0x7f61dc0d0de0 legacy=0x7f61dc0b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61da7fc640 1 -- 172.21.15.146:0/1243346321 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc06cba0 con 0x7f61dc0bb450 2024-04-25T15:20:16.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61d9ffb640 1 -- 172.21.15.146:0/1243346321 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3534572848 0 0) 0x7f61dc06dea0 con 0x7f61dc0d0de0 2024-04-25T15:20:16.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.270+0000 7f61daffd640 1 --2- 172.21.15.146:0/1243346321 >> v2:172.21.15.177:3300/0 conn(0x7f61dc163c00 0x7f61dc06d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:16.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.271+0000 7f61da7fc640 1 --2- 172.21.15.146:0/1243346321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc0bb450 0x7f61dc0d08a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f61dc08e920 tx=0x7f61d002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c587044b3048425 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.271+0000 7f61d9ffb640 1 -- 172.21.15.146:0/1243346321 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0037070 con 0x7f61dc0bb450 2024-04-25T15:20:16.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.271+0000 7f61d9ffb640 1 -- 172.21.15.146:0/1243346321 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61d0032440 con 0x7f61dc0bb450 2024-04-25T15:20:16.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.271+0000 7f61d9ffb640 1 -- 172.21.15.146:0/1243346321 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d002ae60 con 0x7f61dc0bb450 2024-04-25T15:20:16.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.271+0000 7f61e14f9640 1 -- 172.21.15.146:0/1243346321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc0bb450 msgr2=0x7f61dc0d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.271+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1243346321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc0bb450 0x7f61dc0d08a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f61dc08e920 tx=0x7f61d002a110 comp rx=0 tx=0).stop 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.272+0000 7f61e14f9640 1 -- 172.21.15.146:0/1243346321 shutdown_connections 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.272+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1243346321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc0bb450 0x7f61dc0d08a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.272+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1243346321 >> v2:172.21.15.177:3300/0 conn(0x7f61dc163c00 0x7f61dc06d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.272+0000 7f61e14f9640 1 -- 172.21.15.146:0/1243346321 >> 172.21.15.146:0/1243346321 conn(0x7f61dc0b7970 msgr2=0x7f61dc0b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.272+0000 7f61e14f9640 1 -- 172.21.15.146:0/1243346321 shutdown_connections 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.272+0000 7f61e14f9640 1 -- 172.21.15.146:0/1243346321 wait complete. 2024-04-25T15:20:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.274+0000 7f61e14f9640 1 Processor -- start 2024-04-25T15:20:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.274+0000 7f61e14f9640 1 -- start start 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61e14f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61dc0bb450 0x7f61dc0caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61e14f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61dc0d0de0 0x7f61dc0cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61e14f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 0x7f61dc1f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61e14f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc06c550 con 0x7f61dc163c00 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61e14f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc06d160 con 0x7f61dc0bb450 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61e14f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f61dc0bb860 con 0x7f61dc0d0de0 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 0x7f61dc1f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 0x7f61dc1f5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60432/0 (socket says 172.21.15.146:60432) 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 -- 172.21.15.146:0/1691076867 learned_addr learned my addr 172.21.15.146:0/1691076867 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61daffd640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61dc0bb450 0x7f61dc0caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61da7fc640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61dc0d0de0 0x7f61dc0cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 -- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61dc0d0de0 msgr2=0x7f61dc0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61dc0d0de0 0x7f61dc0cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 -- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61dc0bb450 msgr2=0x7f61dc0caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61dc0bb450 0x7f61dc0caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.275+0000 7f61db7fe640 1 -- 172.21.15.146:0/1691076867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc0c9990 con 0x7f61dc163c00 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.276+0000 7f61db7fe640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 0x7f61dc1f5a30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61d400c410 tx=0x7f61d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.276+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d40074e0 con 0x7f61dc163c00 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.276+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61d40047e0 con 0x7f61dc163c00 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.276+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61dc0ccbe0 con 0x7f61dc163c00 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.276+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4004bd0 con 0x7f61dc163c00 2024-04-25T15:20:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.276+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61dc0ca0e0 con 0x7f61dc163c00 2024-04-25T15:20:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.277+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61dc0c9b60 con 0x7f61dc163c00 2024-04-25T15:20:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.277+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f61d4011070 con 0x7f61dc163c00 2024-04-25T15:20:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.278+0000 7f61c37fe640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f61c80744e0 0x7f61c80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.278+0000 7f61daffd640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f61c80744e0 0x7f61c80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.278+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f61d4095100 con 0x7f61dc163c00 2024-04-25T15:20:16.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.278+0000 7f61daffd640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f61c80744e0 0x7f61c80769f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f61dc0b5130 tx=0x7f61cc0039f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.283+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61dc0c9b60 con 0x7f61dc163c00 2024-04-25T15:20:16.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.459+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f61dc06a0e0 con 0x7f61dc163c00 2024-04-25T15:20:16.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.459+0000 7f61c37fe640 1 -- 172.21.15.146:0/1691076867 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f61d405f550 con 0x7f61dc163c00 2024-04-25T15:20:16.461 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:16.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f61c80744e0 msgr2=0x7f61c80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f61c80744e0 0x7f61c80769f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f61dc0b5130 tx=0x7f61cc0039f0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 msgr2=0x7f61dc1f5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 0x7f61dc1f5a30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61d400c410 tx=0x7f61d400c8d0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 shutdown_connections 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f61c80744e0 0x7f61c80769f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61dc163c00 0x7f61dc1f5a30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61dc0d0de0 0x7f61dc0cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 --2- 172.21.15.146:0/1691076867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61dc0bb450 0x7f61dc0caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.461+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 >> 172.21.15.146:0/1691076867 conn(0x7f61dc0b7970 msgr2=0x7f61dc0b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.462+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 shutdown_connections 2024-04-25T15:20:16.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.462+0000 7f61e14f9640 1 -- 172.21.15.146:0/1691076867 wait complete. 2024-04-25T15:20:16.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:16.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.580+0000 7fe41e546640 1 Processor -- start 2024-04-25T15:20:16.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.580+0000 7fe41e546640 1 -- start start 2024-04-25T15:20:16.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.581+0000 7fe41e546640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe4181691d0 0x7fe41816b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.581+0000 7fe41e546640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4180b8eb0 con 0x7fe4180b2ba0 2024-04-25T15:20:16.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.581+0000 7fe41e546640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe41808ce00 con 0x7fe41806ee60 2024-04-25T15:20:16.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.581+0000 7fe41e546640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe418161840 con 0x7fe4181691d0 2024-04-25T15:20:16.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.581+0000 7fe4177fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe4180b2ba0 0x7fe4181645c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40690/0 (socket says 172.21.15.146:40690) 2024-04-25T15:20:16.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.581+0000 7fe4177fe640 1 -- 172.21.15.146:0/609569014 learned_addr learned my addr 172.21.15.146:0/609569014 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe41cae3640 1 --2- 172.21.15.146:0/609569014 >> v2:172.21.15.177:3300/0 conn(0x7fe4181691d0 0x7fe41816b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983125788 0 0) 0x7fe4180b8eb0 con 0x7fe4180b2ba0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe400003520 con 0x7fe4180b2ba0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2935086788 0 0) 0x7fe400003520 con 0x7fe4180b2ba0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 >> v2:172.21.15.177:3300/0 conn(0x7fe4181691d0 msgr2=0x7fe41816b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 --2- 172.21.15.146:0/609569014 >> v2:172.21.15.177:3300/0 conn(0x7fe4181691d0 0x7fe41816b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 >> v1:172.21.15.177:6789/0 conn(0x7fe41806ee60 legacy=0x7fe418163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41808d0e0 con 0x7fe4180b2ba0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fe40c001670 con 0x7fe4180b2ba0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 >> v1:172.21.15.146:6789/0 conn(0x7fe4180b2ba0 legacy=0x7fe4181645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4000039b0 0x7fe400003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe400004800 0x7fe400004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe418161840 con 0x7fe4000039b0 2024-04-25T15:20:16.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe40000c160 con 0x7fe400004800 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe40000c360 con 0x7fe4000051c0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe417fff640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4000039b0 0x7fe400003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe4177fe640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe400004800 0x7fe400004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe41cae3640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe417fff640 1 -- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4000051c0 msgr2=0x7fe40000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe417fff640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe417fff640 1 -- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe400004800 msgr2=0x7fe400004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe417fff640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe400004800 0x7fe400004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe417fff640 1 -- 172.21.15.146:0/609569014 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe400003520 con 0x7fe4000039b0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.582+0000 7fe41cae3640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4000051c0 0x7fe40000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.583+0000 7fe41cae3640 1 -- 172.21.15.146:0/609569014 reap_dead start 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.583+0000 7fe417fff640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4000039b0 0x7fe400003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe408001770 tx=0x7fe400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64346d26cb03efa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.583+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4080013c0 con 0x7fe4000039b0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.583+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4080052d0 con 0x7fe4000039b0 2024-04-25T15:20:16.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.583+0000 7fe416ffd640 1 -- 172.21.15.146:0/609569014 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40802a7f0 con 0x7fe4000039b0 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 -- 172.21.15.146:0/609569014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4000039b0 msgr2=0x7fe400003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4000039b0 0x7fe400003e20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe408001770 tx=0x7fe400005830 comp rx=0 tx=0).stop 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 -- 172.21.15.146:0/609569014 shutdown_connections 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 --2- 172.21.15.146:0/609569014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4000039b0 0x7fe400003e20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 -- 172.21.15.146:0/609569014 >> 172.21.15.146:0/609569014 conn(0x7fe4180b52e0 msgr2=0x7fe4180b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 -- 172.21.15.146:0/609569014 shutdown_connections 2024-04-25T15:20:16.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.584+0000 7fe41e546640 1 -- 172.21.15.146:0/609569014 wait complete. 2024-04-25T15:20:16.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.586+0000 7fe41e546640 1 Processor -- start 2024-04-25T15:20:16.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.586+0000 7fe41e546640 1 -- start start 2024-04-25T15:20:16.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe41e546640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4000039b0 0x7fe418153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe41e546640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 0x7fe4180d2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe41e546640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4180d2570 0x7fe4181619c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe41e546640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe41806f440 con 0x7fe418153bf0 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe41e546640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe41808d730 con 0x7fe4000039b0 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe41e546640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe41808dbc0 con 0x7fe4180d2570 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe4177fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 0x7fe4180d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe4177fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 0x7fe4180d2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60450/0 (socket says 172.21.15.146:60450) 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.587+0000 7fe4177fe640 1 -- 172.21.15.146:0/1529798799 learned_addr learned my addr 172.21.15.146:0/1529798799 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe417fff640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4000039b0 0x7fe418153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe41cae3640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4180d2570 0x7fe4181619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe4177fe640 1 -- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4180d2570 msgr2=0x7fe4181619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe4177fe640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4180d2570 0x7fe4181619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe4177fe640 1 -- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4000039b0 msgr2=0x7fe418153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe4177fe640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4000039b0 0x7fe418153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe4177fe640 1 -- 172.21.15.146:0/1529798799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4180d2980 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe41cae3640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4180d2570 0x7fe4181619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe4177fe640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 0x7fe4180d2030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe40c004fa0 tx=0x7fe40c0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c00f720 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe40c018070 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4180b4e90 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c00fb20 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.588+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4181fa400 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.589+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe41806a0e0 con 0x7fe418153bf0 2024-04-25T15:20:16.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.590+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe40c00fcc0 con 0x7fe418153bf0 2024-04-25T15:20:16.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.590+0000 7fe414ff9640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4040744b0 0x7fe4040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.590+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe40c095410 con 0x7fe418153bf0 2024-04-25T15:20:16.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.590+0000 7fe417fff640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4040744b0 0x7fe4040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.591+0000 7fe417fff640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4040744b0 0x7fe4040769c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe408032000 tx=0x7fe40802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.595+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4180d2980 con 0x7fe418153bf0 2024-04-25T15:20:16.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.805+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4180b4c20 con 0x7fe418153bf0 2024-04-25T15:20:16.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.806+0000 7fe414ff9640 1 -- 172.21.15.146:0/1529798799 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7fe40c05f980 con 0x7fe418153bf0 2024-04-25T15:20:16.808 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:16.808 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4040744b0 msgr2=0x7fe4040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4040744b0 0x7fe4040769c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe408032000 tx=0x7fe40802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 msgr2=0x7fe4180d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 0x7fe4180d2030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe40c004fa0 tx=0x7fe40c0074f0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 shutdown_connections 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4040744b0 0x7fe4040769c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4180d2570 0x7fe4181619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe418153bf0 0x7fe4180d2030 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 --2- 172.21.15.146:0/1529798799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4000039b0 0x7fe418153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 >> 172.21.15.146:0/1529798799 conn(0x7fe4180b52e0 msgr2=0x7fe4180cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:16.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 shutdown_connections 2024-04-25T15:20:16.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.808+0000 7fe41e546640 1 -- 172.21.15.146:0/1529798799 wait complete. 2024-04-25T15:20:16.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:16.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.927+0000 7fdfe729f640 1 Processor -- start 2024-04-25T15:20:16.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.928+0000 7fdfe729f640 1 -- start start 2024-04-25T15:20:16.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe729f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdfe006ee60 0x7fdfe00b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe729f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfe00b8eb0 con 0x7fdfe0163eb0 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe729f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfe008ce00 con 0x7fdfe01649d0 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe729f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fdfe0161840 con 0x7fdfe006ee60 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe483a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdfe0163eb0 0x7fdfe01642c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40702/0 (socket says 172.21.15.146:40702) 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe483a640 1 -- 172.21.15.146:0/48663773 learned_addr learned my addr 172.21.15.146:0/48663773 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfe503b640 1 --2- 172.21.15.146:0/48663773 >> v2:172.21.15.177:3300/0 conn(0x7fdfe006ee60 0x7fdfe00b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522123936 0 0) 0x7fdfe00b8eb0 con 0x7fdfe0163eb0 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.929+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc003520 con 0x7fdfe0163eb0 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2116718673 0 0) 0x7fdfcc003520 con 0x7fdfe0163eb0 2024-04-25T15:20:16.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 >> v2:172.21.15.177:3300/0 conn(0x7fdfe006ee60 msgr2=0x7fdfe00b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 --2- 172.21.15.146:0/48663773 >> v2:172.21.15.177:3300/0 conn(0x7fdfe006ee60 0x7fdfe00b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 >> v1:172.21.15.177:6789/0 conn(0x7fdfe01649d0 legacy=0x7fdfe00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe008d0e0 con 0x7fdfe0163eb0 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fdfd0001670 con 0x7fdfe0163eb0 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 >> v1:172.21.15.146:6789/0 conn(0x7fdfe0163eb0 legacy=0x7fdfe01642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc0039b0 0x7fdfcc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc004800 0x7fdfcc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfcc0051c0 0x7fdfcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe0161840 con 0x7fdfcc004800 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc00c160 con 0x7fdfcc0039b0 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdfcc00c360 con 0x7fdfcc0051c0 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe483a640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc004800 0x7fdfcc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe503b640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc0039b0 0x7fdfcc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe583c640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfcc0051c0 0x7fdfcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe483a640 1 -- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfcc0051c0 msgr2=0x7fdfcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe483a640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfcc0051c0 0x7fdfcc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe483a640 1 -- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc0039b0 msgr2=0x7fdfcc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe483a640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc0039b0 0x7fdfcc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe483a640 1 -- 172.21.15.146:0/48663773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc003520 con 0x7fdfcc004800 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe583c640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfcc0051c0 0x7fdfcc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.930+0000 7fdfe583c640 1 -- 172.21.15.146:0/48663773 reap_dead start 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.931+0000 7fdfe483a640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc004800 0x7fdfcc004c10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c240 tx=0x7fdfcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a28c87f6e638fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.931+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0007d10 con 0x7fdfcc004800 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.931+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd0005ad0 con 0x7fdfcc004800 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.931+0000 7fdfd7fff640 1 -- 172.21.15.146:0/48663773 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0007070 con 0x7fdfcc004800 2024-04-25T15:20:16.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 -- 172.21.15.146:0/48663773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc004800 msgr2=0x7fdfcc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc004800 0x7fdfcc004c10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c240 tx=0x7fdfcc005830 comp rx=0 tx=0).stop 2024-04-25T15:20:16.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 -- 172.21.15.146:0/48663773 shutdown_connections 2024-04-25T15:20:16.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 --2- 172.21.15.146:0/48663773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc004800 0x7fdfcc004c10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 -- 172.21.15.146:0/48663773 >> 172.21.15.146:0/48663773 conn(0x7fdfe00b52e0 msgr2=0x7fdfe00b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:16.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 -- 172.21.15.146:0/48663773 shutdown_connections 2024-04-25T15:20:16.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.932+0000 7fdfe729f640 1 -- 172.21.15.146:0/48663773 wait complete. 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.934+0000 7fdfe729f640 1 Processor -- start 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.934+0000 7fdfe729f640 1 -- start start 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe729f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc004800 0x7fdfe01537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe729f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfe00b6c30 0x7fdfe00b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe729f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 0x7fdfe01f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe729f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe008d730 con 0x7fdfe00b7580 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe729f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe006f440 con 0x7fdfcc004800 2024-04-25T15:20:16.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe729f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe008dbc0 con 0x7fdfe00b6c30 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 0x7fdfe01f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 0x7fdfe01f9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60460/0 (socket says 172.21.15.146:60460) 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 -- 172.21.15.146:0/640844448 learned_addr learned my addr 172.21.15.146:0/640844448 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe503b640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc004800 0x7fdfe01537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe483a640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfe00b6c30 0x7fdfe00b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 -- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfe00b6c30 msgr2=0x7fdfe00b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfe00b6c30 0x7fdfe00b7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 -- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc004800 msgr2=0x7fdfe01537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc004800 0x7fdfe01537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 -- 172.21.15.146:0/640844448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe0153cf0 con 0x7fdfe00b7580 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe503b640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc004800 0x7fdfe01537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe483a640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfe00b6c30 0x7fdfe00b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:16.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.935+0000 7fdfe583c640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 0x7fdfe01f9b70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00d37a0 tx=0x7fdfd800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.936+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8003970 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.936+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd8018070 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.936+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8003d50 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.936+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe0162bb0 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.936+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe0163170 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.937+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe00b2ef0 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.937+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdfd80050d0 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.938+0000 7fdfd5ffb640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdfc40744e0 0x7fdfc40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.938+0000 7fdfe503b640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdfc40744e0 0x7fdfc40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.938+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdfd8094fd0 con 0x7fdfe00b7580 2024-04-25T15:20:16.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.938+0000 7fdfe503b640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdfc40744e0 0x7fdfc40769f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009b20 tx=0x7fdfdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:16.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:16.943+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdfd8022020 con 0x7fdfe00b7580 2024-04-25T15:20:17.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.121+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdfe006a0e0 con 0x7fdfe00b7580 2024-04-25T15:20:17.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.122+0000 7fdfd5ffb640 1 -- 172.21.15.146:0/640844448 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7fdfd805f420 con 0x7fdfe00b7580 2024-04-25T15:20:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:17.124 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:17.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdfc40744e0 msgr2=0x7fdfc40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdfc40744e0 0x7fdfc40769f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009b20 tx=0x7fdfdc008040 comp rx=0 tx=0).stop 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 msgr2=0x7fdfe01f9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 0x7fdfe01f9b70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00d37a0 tx=0x7fdfd800eb90 comp rx=0 tx=0).stop 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 shutdown_connections 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdfc40744e0 0x7fdfc40769f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfe00b7580 0x7fdfe01f9b70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdfe00b6c30 0x7fdfe00b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 --2- 172.21.15.146:0/640844448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfcc004800 0x7fdfe01537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.124+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 >> 172.21.15.146:0/640844448 conn(0x7fdfe00b52e0 msgr2=0x7fdfe00ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.125+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 shutdown_connections 2024-04-25T15:20:17.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.125+0000 7fdfe729f640 1 -- 172.21.15.146:0/640844448 wait complete. 2024-04-25T15:20:17.138 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:17.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:17.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.248+0000 7fb3f1c67640 1 Processor -- start 2024-04-25T15:20:17.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.248+0000 7fb3f1c67640 1 -- start start 2024-04-25T15:20:17.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.249+0000 7fb3f1c67640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb3ec0b2cd0 0x7fb3ec0cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.249+0000 7fb3f1c67640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3ec0d2440 0x7fb3ec071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.249+0000 7fb3f1c67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ec0cf7d0 con 0x7fb3ec0d1920 2024-04-25T15:20:17.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.249+0000 7fb3f1c67640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ec08da90 con 0x7fb3ec0d2440 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.249+0000 7fb3f1c67640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ec0d0070 con 0x7fb3ec0b2cd0 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3ebfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3ec0d2440 0x7fb3ec071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3eaffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb3ec0d1920 0x7fb3ec0d1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40704/0 (socket says 172.21.15.146:40704) 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3ebfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3ec0d2440 0x7fb3ec071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60474/0 (socket says 172.21.15.146:60474) 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3eaffd640 1 -- 172.21.15.146:0/4054313693 learned_addr learned my addr 172.21.15.146:0/4054313693 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3eb7fe640 1 --2- 172.21.15.146:0/4054313693 >> v2:172.21.15.177:3300/0 conn(0x7fb3ec0b2cd0 0x7fb3ec0cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4054313693 >> v2:172.21.15.177:3300/0 conn(0x7fb3ec0b2cd0 msgr2=0x7fb3ec0cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3ebfff640 1 --2- 172.21.15.146:0/4054313693 >> v2:172.21.15.177:3300/0 conn(0x7fb3ec0b2cd0 0x7fb3ec0cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4054313693 >> v1:172.21.15.146:6789/0 conn(0x7fb3ec0d1920 legacy=0x7fb3ec0d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4054313693 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec0cfab0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.250+0000 7fb3eb7fe640 1 --2- 172.21.15.146:0/4054313693 >> v2:172.21.15.177:3300/0 conn(0x7fb3ec0b2cd0 0x7fb3ec0cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:17.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3ebfff640 1 --2- 172.21.15.146:0/4054313693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec071c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0b6040 tx=0x7fb3e402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cab8f86140bbca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3ea7fc640 1 -- 172.21.15.146:0/4054313693 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4007740 con 0x7fb3ec0d2440 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3ea7fc640 1 -- 172.21.15.146:0/4054313693 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e40078e0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3ea7fc640 1 -- 172.21.15.146:0/4054313693 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4007cb0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4054313693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 msgr2=0x7fb3ec071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4054313693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec071c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0b6040 tx=0x7fb3e402a5f0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4054313693 shutdown_connections 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4054313693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec071c70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4054313693 >> v2:172.21.15.177:3300/0 conn(0x7fb3ec0b2cd0 0x7fb3ec0cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4054313693 >> 172.21.15.146:0/4054313693 conn(0x7fb3ec0b52e0 msgr2=0x7fb3ec0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4054313693 shutdown_connections 2024-04-25T15:20:17.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.251+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4054313693 wait complete. 2024-04-25T15:20:17.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.253+0000 7fb3f1c67640 1 Processor -- start 2024-04-25T15:20:17.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.253+0000 7fb3f1c67640 1 -- start start 2024-04-25T15:20:17.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3ec0b2cd0 0x7fb3ec154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3ec0d1920 0x7fb3ec0a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec0721b0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec0717e0 con 0x7fb3ec0b2cd0 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ec072330 con 0x7fb3ec0d1920 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec0cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60480/0 (socket says 172.21.15.146:60480) 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4224833676 learned_addr learned my addr 172.21.15.146:0/4224833676 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3eb7fe640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3ec0b2cd0 0x7fb3ec154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3eaffd640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3ec0d1920 0x7fb3ec0a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3ec0d1920 msgr2=0x7fb3ec0a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3ec0d1920 0x7fb3ec0a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3ec0b2cd0 msgr2=0x7fb3ec154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3ec0b2cd0 0x7fb3ec154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 -- 172.21.15.146:0/4224833676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec09ff30 con 0x7fb3ec0d2440 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3eb7fe640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3ec0b2cd0 0x7fb3ec154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:17.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3eaffd640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3ec0d1920 0x7fb3ec0a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3ebfff640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec0cb290 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3e402abe0 tx=0x7fb3e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e403f070 con 0x7fb3ec0d2440 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e4031bc0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ec0a0210 con 0x7fb3ec0d2440 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4044460 con 0x7fb3ec0d2440 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.255+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ec0a07d0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.257+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ec09fc80 con 0x7fb3ec0d2440 2024-04-25T15:20:17.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.257+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb3e403b410 con 0x7fb3ec0d2440 2024-04-25T15:20:17.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.258+0000 7fb3d3fff640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3d80744b0 0x7fb3d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.258+0000 7fb3eb7fe640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3d80744b0 0x7fb3d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.258+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb3e40b6e00 con 0x7fb3ec0d2440 2024-04-25T15:20:17.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.258+0000 7fb3eb7fe640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3d80744b0 0x7fb3d80769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0ce350 tx=0x7fb3dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.262+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3e4080c90 con 0x7fb3ec0d2440 2024-04-25T15:20:17.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.437+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb3ec06a0e0 con 0x7fb3ec0d2440 2024-04-25T15:20:17.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.438+0000 7fb3d3fff640 1 -- 172.21.15.146:0/4224833676 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fb3e4084f90 con 0x7fb3ec0d2440 2024-04-25T15:20:17.440 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3d80744b0 msgr2=0x7fb3d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3d80744b0 0x7fb3d80769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0ce350 tx=0x7fb3dc008040 comp rx=0 tx=0).stop 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 msgr2=0x7fb3ec0cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec0cb290 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb3e402abe0 tx=0x7fb3e4004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 shutdown_connections 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3d80744b0 0x7fb3d80769c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec0d2440 0x7fb3ec0cb290 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3ec0d1920 0x7fb3ec0a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 --2- 172.21.15.146:0/4224833676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3ec0b2cd0 0x7fb3ec154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 >> 172.21.15.146:0/4224833676 conn(0x7fb3ec0b52e0 msgr2=0x7fb3ec0b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:17.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 shutdown_connections 2024-04-25T15:20:17.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.440+0000 7fb3f1c67640 1 -- 172.21.15.146:0/4224833676 wait complete. 2024-04-25T15:20:17.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:17.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.562+0000 7ff4a6fa8640 1 Processor -- start 2024-04-25T15:20:17.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.562+0000 7ff4a6fa8640 1 -- start start 2024-04-25T15:20:17.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff4a6fa8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a006dc70 0x7ff4a00b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff4a6fa8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a00b0520 con 0x7ff4a00d10b0 2024-04-25T15:20:17.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff4a6fa8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a0070d90 con 0x7ff4a0070500 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff4a6fa8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a00b0850 con 0x7ff4a006dc70 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff4a5545640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff4a00d10b0 0x7ff4a00b4610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40720/0 (socket says 172.21.15.146:40720) 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff4a5545640 1 -- 172.21.15.146:0/145563238 learned_addr learned my addr 172.21.15.146:0/145563238 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49ffff640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a006dc70 0x7ff4a00b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932637177 0 0) 0x7ff4a00b0520 con 0x7ff4a00d10b0 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff488003520 con 0x7ff4a00d10b0 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3382456188 0 0) 0x7ff4a0070d90 con 0x7ff4a0070500 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4a00b0520 con 0x7ff4a0070500 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 69660548 0 0) 0x7ff488003520 con 0x7ff4a00d10b0 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a006dc70 msgr2=0x7ff4a00b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.564+0000 7ff49f7fe640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a006dc70 0x7ff4a00b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 >> v1:172.21.15.177:6789/0 conn(0x7ff4a0070500 legacy=0x7ff4a006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a0071070 con 0x7ff4a00d10b0 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7ff498001a10 con 0x7ff4a00d10b0 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 >> v1:172.21.15.146:6789/0 conn(0x7ff4a00d10b0 legacy=0x7ff4a00b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff488003b70 0x7ff488003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488004800 0x7ff488004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488005150 0x7ff48800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a00b0850 con 0x7ff488005150 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff48800bfb0 con 0x7ff488004800 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff48800c1b0 con 0x7ff488003b70 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a5545640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488005150 0x7ff48800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a4d44640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff488003b70 0x7ff488003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff49ffff640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488004800 0x7ff488004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a5545640 1 -- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff488003b70 msgr2=0x7ff488003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a5545640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff488003b70 0x7ff488003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a5545640 1 -- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488004800 msgr2=0x7ff488004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a5545640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488004800 0x7ff488004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.565+0000 7ff4a5545640 1 -- 172.21.15.146:0/145563238 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff488003520 con 0x7ff488005150 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.566+0000 7ff4a5545640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488005150 0x7ff48800b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00b06f0 tx=0x7ff488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae8237fd1489681 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.566+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49800a350 con 0x7ff488005150 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.566+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff498031070 con 0x7ff488005150 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.566+0000 7ff49f7fe640 1 -- 172.21.15.146:0/145563238 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49802dcd0 con 0x7ff488005150 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/145563238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488005150 msgr2=0x7ff48800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/145563238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488005150 0x7ff48800b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00b06f0 tx=0x7ff488005830 comp rx=0 tx=0).stop 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a5545640 1 -- 172.21.15.146:0/145563238 reap_dead start 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/145563238 shutdown_connections 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/145563238 >> 172.21.15.146:0/145563238 conn(0x7ff4a00d1c10 msgr2=0x7ff4a00733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/145563238 shutdown_connections 2024-04-25T15:20:17.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.567+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/145563238 wait complete. 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.568+0000 7ff4a6fa8640 1 Processor -- start 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.568+0000 7ff4a6fa8640 1 -- start start 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a6fa8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488005150 0x7ff4a0153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a6fa8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a0141b20 0x7ff4a0143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a6fa8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 0x7ff4a00cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a6fa8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a006ddf0 con 0x7ff4a01439a0 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a6fa8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a006dc70 con 0x7ff488005150 2024-04-25T15:20:17.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a6fa8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a00b0ce0 con 0x7ff4a0141b20 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 0x7ff4a00cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 0x7ff4a00cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60488/0 (socket says 172.21.15.146:60488) 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 -- 172.21.15.146:0/1750436195 learned_addr learned my addr 172.21.15.146:0/1750436195 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a4d44640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488005150 0x7ff4a0153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff49ffff640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a0141b20 0x7ff4a0143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 -- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a0141b20 msgr2=0x7ff4a0143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a0141b20 0x7ff4a0143460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 -- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488005150 msgr2=0x7ff4a0153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488005150 0x7ff4a0153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.570+0000 7ff4a5545640 1 -- 172.21.15.146:0/1750436195 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a0148110 con 0x7ff4a01439a0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.571+0000 7ff4a5545640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 0x7ff4a00cdff0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff498000f80 tx=0x7ff4a0071240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.571+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49800a350 con 0x7ff4a01439a0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.571+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff498004b60 con 0x7ff4a01439a0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.571+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a01423e0 con 0x7ff4a01439a0 2024-04-25T15:20:17.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.571+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498001ec0 con 0x7ff4a01439a0 2024-04-25T15:20:17.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.571+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a01426a0 con 0x7ff4a01439a0 2024-04-25T15:20:17.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.572+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a006a0e0 con 0x7ff4a01439a0 2024-04-25T15:20:17.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.572+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff498036470 con 0x7ff4a01439a0 2024-04-25T15:20:17.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.573+0000 7ff49d7fa640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff48c0744b0 0x7ff48c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.573+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff4980b7850 con 0x7ff4a01439a0 2024-04-25T15:20:17.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.573+0000 7ff4a4d44640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff48c0744b0 0x7ff48c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.573+0000 7ff4a4d44640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff48c0744b0 0x7ff48c0769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff490000c00 tx=0x7ff490009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.578+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4980817a0 con 0x7ff4a01439a0 2024-04-25T15:20:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.760+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff4a00cddd0 con 0x7ff4a01439a0 2024-04-25T15:20:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.761+0000 7ff49d7fa640 1 -- 172.21.15.146:0/1750436195 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ff498085aa0 con 0x7ff4a01439a0 2024-04-25T15:20:17.763 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:17.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff48c0744b0 msgr2=0x7ff48c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff48c0744b0 0x7ff48c0769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff490000c00 tx=0x7ff490009040 comp rx=0 tx=0).stop 2024-04-25T15:20:17.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 msgr2=0x7ff4a00cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 0x7ff4a00cdff0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff498000f80 tx=0x7ff4a0071240 comp rx=0 tx=0).stop 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 shutdown_connections 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff48c0744b0 0x7ff48c0769c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a01439a0 0x7ff4a00cdff0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4a0141b20 0x7ff4a0143460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 --2- 172.21.15.146:0/1750436195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488005150 0x7ff4a0153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 >> 172.21.15.146:0/1750436195 conn(0x7ff4a00d1c10 msgr2=0x7ff4a0157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 shutdown_connections 2024-04-25T15:20:17.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.763+0000 7ff4a6fa8640 1 -- 172.21.15.146:0/1750436195 wait complete. 2024-04-25T15:20:17.776 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:17.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.886+0000 7f6f6b820640 1 Processor -- start 2024-04-25T15:20:17.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.886+0000 7f6f6b820640 1 -- start start 2024-04-25T15:20:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f6b820640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f640b8f30 0x7f6f640cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f6b820640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f6408d590 con 0x7f6f640b4ca0 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f6b820640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f640d27d0 con 0x7f6f6408dd50 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f6b820640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6f640b9340 con 0x7f6f640b8f30 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f69dbd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f640b8f30 0x7f6f640cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f69dbd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f640b8f30 0x7f6f640cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60494/0 (socket says 172.21.15.146:60494) 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f69dbd640 1 -- 172.21.15.146:0/4289477108 learned_addr learned my addr 172.21.15.146:0/4289477108 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.887+0000 7f6f68dbb640 1 --1- 172.21.15.146:0/4289477108 >> v1:172.21.15.146:6789/0 conn(0x7f6f640b4ca0 0x7f6f640ce6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40734/0 (socket says 172.21.15.146:40734) 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f69dbd640 1 -- 172.21.15.146:0/4289477108 >> v1:172.21.15.177:6789/0 conn(0x7f6f6408dd50 legacy=0x7f6f640d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f69dbd640 1 -- 172.21.15.146:0/4289477108 >> v1:172.21.15.146:6789/0 conn(0x7f6f640b4ca0 legacy=0x7f6f640ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f5bfff640 1 -- 172.21.15.146:0/4289477108 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192752258 0 0) 0x7f6f6408d590 con 0x7f6f640b4ca0 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f69dbd640 1 -- 172.21.15.146:0/4289477108 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f640b48b0 con 0x7f6f640b8f30 2024-04-25T15:20:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f69dbd640 1 --2- 172.21.15.146:0/4289477108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f640b8f30 0x7f6f640cee00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009870 tx=0x7f6f5c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e1c7df1c8dab37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f5bfff640 1 -- 172.21.15.146:0/4289477108 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c037070 con 0x7f6f640b8f30 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f5bfff640 1 -- 172.21.15.146:0/4289477108 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5c02ae50 con 0x7f6f640b8f30 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.888+0000 7f6f5bfff640 1 -- 172.21.15.146:0/4289477108 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c032860 con 0x7f6f640b8f30 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 -- 172.21.15.146:0/4289477108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f640b8f30 msgr2=0x7f6f640cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 --2- 172.21.15.146:0/4289477108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f640b8f30 0x7f6f640cee00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009870 tx=0x7f6f5c02a300 comp rx=0 tx=0).stop 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 -- 172.21.15.146:0/4289477108 shutdown_connections 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 --2- 172.21.15.146:0/4289477108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f640b8f30 0x7f6f640cee00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 -- 172.21.15.146:0/4289477108 >> 172.21.15.146:0/4289477108 conn(0x7f6f640b5430 msgr2=0x7f6f640b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 -- 172.21.15.146:0/4289477108 shutdown_connections 2024-04-25T15:20:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.889+0000 7f6f6b820640 1 -- 172.21.15.146:0/4289477108 wait complete. 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.891+0000 7f6f6b820640 1 Processor -- start 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.891+0000 7f6f6b820640 1 -- start start 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f6b820640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f640b4ca0 0x7f6f640cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f6b820640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f640b8f30 0x7f6f640cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f6b820640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 0x7f6f641f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f6b820640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6f640d2340 con 0x7f6f641f97e0 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f6b820640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6408d8c0 con 0x7f6f640b4ca0 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f6b820640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6f640d24c0 con 0x7f6f640b8f30 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f69dbd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 0x7f6f641f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f69dbd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 0x7f6f641f9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60496/0 (socket says 172.21.15.146:60496) 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f69dbd640 1 -- 172.21.15.146:0/2038653661 learned_addr learned my addr 172.21.15.146:0/2038653661 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f68dbb640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f640b8f30 0x7f6f640cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.892+0000 7f6f695bc640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f640b4ca0 0x7f6f640cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f69dbd640 1 -- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f640b8f30 msgr2=0x7f6f640cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f69dbd640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f640b8f30 0x7f6f640cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f69dbd640 1 -- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f640b4ca0 msgr2=0x7f6f640cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f69dbd640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f640b4ca0 0x7f6f640cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f69dbd640 1 -- 172.21.15.146:0/2038653661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f641fa130 con 0x7f6f641f97e0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f695bc640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f640b4ca0 0x7f6f640cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f69dbd640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 0x7f6f641f9bf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009870 tx=0x7f6f5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c037040 con 0x7f6f641f97e0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5c004250 con 0x7f6f641f97e0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f641d6790 con 0x7f6f641f97e0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c003610 con 0x7f6f641f97e0 2024-04-25T15:20:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.893+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f641d6d50 con 0x7f6f641f97e0 2024-04-25T15:20:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.894+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f6406a0e0 con 0x7f6f641f97e0 2024-04-25T15:20:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.895+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6f5c03a070 con 0x7f6f641f97e0 2024-04-25T15:20:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.895+0000 7f6f59ffb640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f480744b0 0x7f6f480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:17.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.895+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6f5c0b5d60 con 0x7f6f641f97e0 2024-04-25T15:20:17.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.895+0000 7f6f695bc640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f480744b0 0x7f6f480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:17.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.896+0000 7f6f695bc640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f480744b0 0x7f6f480769c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6f600012e0 tx=0x7f6f60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:17.900+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f641fa130 con 0x7f6f641f97e0 2024-04-25T15:20:18.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.076+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6f640b6f30 con 0x7f6f641f97e0 2024-04-25T15:20:18.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.076+0000 7f6f59ffb640 1 -- 172.21.15.146:0/2038653661 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f6f5c041020 con 0x7f6f641f97e0 2024-04-25T15:20:18.078 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:18.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f480744b0 msgr2=0x7f6f480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f480744b0 0x7f6f480769c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6f600012e0 tx=0x7f6f60009040 comp rx=0 tx=0).stop 2024-04-25T15:20:18.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 msgr2=0x7f6f641f9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 0x7f6f641f9bf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009870 tx=0x7f6f5c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 shutdown_connections 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f480744b0 0x7f6f480769c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f641f97e0 0x7f6f641f9bf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f640b8f30 0x7f6f640cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 --2- 172.21.15.146:0/2038653661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f640b4ca0 0x7f6f640cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 >> 172.21.15.146:0/2038653661 conn(0x7f6f640b5430 msgr2=0x7f6f640b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 shutdown_connections 2024-04-25T15:20:18.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.078+0000 7f6f6b820640 1 -- 172.21.15.146:0/2038653661 wait complete. 2024-04-25T15:20:18.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:18.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.200+0000 7f032817a640 1 Processor -- start 2024-04-25T15:20:18.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.200+0000 7f032817a640 1 -- start start 2024-04-25T15:20:18.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f032817a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03200d2d90 0x7f032009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f032817a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03200b2400 0x7f03200b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f032817a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200b1960 con 0x7f03200ccd40 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f032817a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f03200b3ce0 con 0x7f03200d2d90 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f032817a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f03200b1c00 con 0x7f03200b2400 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03200d2d90 0x7f032009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03200d2d90 0x7f032009bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60508/0 (socket says 172.21.15.146:60508) 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 -- 172.21.15.146:0/1272140008 learned_addr learned my addr 172.21.15.146:0/1272140008 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0326717640 1 --2- 172.21.15.146:0/1272140008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03200b2400 0x7f03200b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 -- 172.21.15.146:0/1272140008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03200b2400 msgr2=0x7f03200b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 --2- 172.21.15.146:0/1272140008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03200b2400 0x7f03200b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 -- 172.21.15.146:0/1272140008 >> v1:172.21.15.177:6789/0 conn(0x7f03200ccd40 legacy=0x7f03200d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.202+0000 7f0325715640 1 -- 172.21.15.146:0/1272140008 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03200b1ee0 con 0x7f03200d2d90 2024-04-25T15:20:18.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.203+0000 7f0325715640 1 --2- 172.21.15.146:0/1272140008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200d2d90 0x7f032009bba0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0310009870 tx=0x7f031002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa151723f508629 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.203+0000 7f0324f14640 1 -- 172.21.15.146:0/1272140008 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310037070 con 0x7f03200d2d90 2024-04-25T15:20:18.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.203+0000 7f0324f14640 1 -- 172.21.15.146:0/1272140008 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0310032440 con 0x7f03200d2d90 2024-04-25T15:20:18.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.203+0000 7f0324f14640 1 -- 172.21.15.146:0/1272140008 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310032810 con 0x7f03200d2d90 2024-04-25T15:20:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 -- 172.21.15.146:0/1272140008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200d2d90 msgr2=0x7f032009bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 --2- 172.21.15.146:0/1272140008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200d2d90 0x7f032009bba0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0310009870 tx=0x7f031002a270 comp rx=0 tx=0).stop 2024-04-25T15:20:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 -- 172.21.15.146:0/1272140008 shutdown_connections 2024-04-25T15:20:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 --2- 172.21.15.146:0/1272140008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f03200b2400 0x7f03200b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 --2- 172.21.15.146:0/1272140008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200d2d90 0x7f032009bba0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 -- 172.21.15.146:0/1272140008 >> 172.21.15.146:0/1272140008 conn(0x7f03200b7400 msgr2=0x7f03200b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 -- 172.21.15.146:0/1272140008 shutdown_connections 2024-04-25T15:20:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.204+0000 7f032817a640 1 -- 172.21.15.146:0/1272140008 wait complete. 2024-04-25T15:20:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.205+0000 7f032817a640 1 Processor -- start 2024-04-25T15:20:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.206+0000 7f032817a640 1 -- start start 2024-04-25T15:20:18.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f032817a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 0x7f0320155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f032817a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03200ccd40 0x7f03200c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f032817a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0320161cf0 0x7f0320162100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f032817a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03200b3ac0 con 0x7f03200b2400 2024-04-25T15:20:18.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f032817a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03200b3940 con 0x7f03200ccd40 2024-04-25T15:20:18.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f032817a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f032009b440 con 0x7f0320161cf0 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 0x7f0320155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 0x7f0320155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60524/0 (socket says 172.21.15.146:60524) 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 -- 172.21.15.146:0/2147094234 learned_addr learned my addr 172.21.15.146:0/2147094234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325715640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03200ccd40 0x7f03200c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0326717640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0320161cf0 0x7f0320162100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 -- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0320161cf0 msgr2=0x7f0320162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0320161cf0 0x7f0320162100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 -- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03200ccd40 msgr2=0x7f03200c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03200ccd40 0x7f03200c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 -- 172.21.15.146:0/2147094234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0320162640 con 0x7f03200b2400 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325715640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03200ccd40 0x7f03200c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f0325f16640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 0x7f0320155d30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f031c00c3c0 tx=0x7f031c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.207+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c017070 con 0x7f03200b2400 2024-04-25T15:20:18.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.208+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031c0052f0 con 0x7f03200b2400 2024-04-25T15:20:18.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.208+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c003680 con 0x7f03200b2400 2024-04-25T15:20:18.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.208+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03200ca260 con 0x7f03200b2400 2024-04-25T15:20:18.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.208+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03200ca820 con 0x7f03200b2400 2024-04-25T15:20:18.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.208+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f032006a0e0 con 0x7f03200b2400 2024-04-25T15:20:18.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.209+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f031c01a070 con 0x7f03200b2400 2024-04-25T15:20:18.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.210+0000 7f03167fc640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03040744e0 0x7f03040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.210+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f031c0949a0 con 0x7f03200b2400 2024-04-25T15:20:18.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.210+0000 7f0325715640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03040744e0 0x7f03040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.210+0000 7f0325715640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03040744e0 0x7f03040769f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0310034000 tx=0x7f031002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.214+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0320162640 con 0x7f03200b2400 2024-04-25T15:20:18.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.387+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f032006f810 con 0x7f03200b2400 2024-04-25T15:20:18.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.388+0000 7f03167fc640 1 -- 172.21.15.146:0/2147094234 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f031c065e20 con 0x7f03200b2400 2024-04-25T15:20:18.390 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03040744e0 msgr2=0x7f03040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03040744e0 0x7f03040769f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0310034000 tx=0x7f031002e000 comp rx=0 tx=0).stop 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 msgr2=0x7f0320155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 0x7f0320155d30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f031c00c3c0 tx=0x7f031c00c7c0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 shutdown_connections 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03040744e0 0x7f03040769f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0320161cf0 0x7f0320162100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03200ccd40 0x7f03200c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 --2- 172.21.15.146:0/2147094234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03200b2400 0x7f0320155d30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 >> 172.21.15.146:0/2147094234 conn(0x7f03200b7400 msgr2=0x7f03200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 shutdown_connections 2024-04-25T15:20:18.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.390+0000 7f032817a640 1 -- 172.21.15.146:0/2147094234 wait complete. 2024-04-25T15:20:18.403 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:18.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.511+0000 7f9bea3ee640 1 Processor -- start 2024-04-25T15:20:18.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.511+0000 7f9bea3ee640 1 -- start start 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.512+0000 7f9bea3ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b6610 0x7f9be40d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.512+0000 7f9bea3ee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9be415ffa0 0x7f9be40b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.512+0000 7f9bea3ee640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9be40b5b90 con 0x7f9be40b5780 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.512+0000 7f9bea3ee640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f9be40b2f50 con 0x7f9be415ffa0 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.512+0000 7f9bea3ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9be40b2ac0 con 0x7f9be40b6610 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be3fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b6610 0x7f9be40d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be37fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9be415ffa0 0x7f9be40b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be898b640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f9be40b5780 0x7f9be4163b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:39072/0 (socket says 172.21.15.146:39072) 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be898b640 1 -- 172.21.15.146:0/1588442363 learned_addr learned my addr 172.21.15.146:0/1588442363 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be2ffd640 1 -- 172.21.15.146:0/1588442363 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1857441655 0 0) 0x7f9bdc001490 con 0x7f9be40b5780 2024-04-25T15:20:18.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be2ffd640 1 -- 172.21.15.146:0/1588442363 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bd0003520 con 0x7f9be40b5780 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be3fff640 1 -- 172.21.15.146:0/1588442363 >> v2:172.21.15.177:3300/0 conn(0x7f9be415ffa0 msgr2=0x7f9be40b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be3fff640 1 --2- 172.21.15.146:0/1588442363 >> v2:172.21.15.177:3300/0 conn(0x7f9be415ffa0 0x7f9be40b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be3fff640 1 -- 172.21.15.146:0/1588442363 >> v1:172.21.15.177:6789/0 conn(0x7f9be40b5780 legacy=0x7f9be4163b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be3fff640 1 -- 172.21.15.146:0/1588442363 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be40d2230 con 0x7f9be40b6610 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.513+0000 7f9be37fe640 1 --2- 172.21.15.146:0/1588442363 >> v2:172.21.15.177:3300/0 conn(0x7f9be415ffa0 0x7f9be40b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.514+0000 7f9be3fff640 1 --2- 172.21.15.146:0/1588442363 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b6610 0x7f9be40d2df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40097a0 tx=0x7f9bd402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daa792dbf49a785e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.514+0000 7f9be2ffd640 1 -- 172.21.15.146:0/1588442363 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4037070 con 0x7f9be40b6610 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.514+0000 7f9be2ffd640 1 -- 172.21.15.146:0/1588442363 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd4032440 con 0x7f9be40b6610 2024-04-25T15:20:18.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.514+0000 7f9be2ffd640 1 -- 172.21.15.146:0/1588442363 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd40327f0 con 0x7f9be40b6610 2024-04-25T15:20:18.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 -- 172.21.15.146:0/1588442363 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b6610 msgr2=0x7f9be40d2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/1588442363 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b6610 0x7f9be40d2df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40097a0 tx=0x7f9bd402a180 comp rx=0 tx=0).stop 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 -- 172.21.15.146:0/1588442363 shutdown_connections 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/1588442363 >> v2:172.21.15.177:3300/0 conn(0x7f9be415ffa0 0x7f9be40b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/1588442363 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b6610 0x7f9be40d2df0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 -- 172.21.15.146:0/1588442363 >> 172.21.15.146:0/1588442363 conn(0x7f9be40d0240 msgr2=0x7f9be40d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 -- 172.21.15.146:0/1588442363 shutdown_connections 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.515+0000 7f9bea3ee640 1 -- 172.21.15.146:0/1588442363 wait complete. 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.516+0000 7f9bea3ee640 1 Processor -- start 2024-04-25T15:20:18.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.517+0000 7f9bea3ee640 1 -- start start 2024-04-25T15:20:18.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9bea3ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b5780 0x7f9be40b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9bea3ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9be40b6610 0x7f9be41f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9bea3ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 0x7f9be41f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9bea3ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9be41603b0 con 0x7f9be415ffa0 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9bea3ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9be40cfd60 con 0x7f9be40b6610 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9bea3ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9be40d1da0 con 0x7f9be40b5780 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 0x7f9be41f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 0x7f9be41f6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60538/0 (socket says 172.21.15.146:60538) 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 -- 172.21.15.146:0/3661090936 learned_addr learned my addr 172.21.15.146:0/3661090936 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be37fe640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9be40b6610 0x7f9be41f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be3fff640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b5780 0x7f9be40b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 -- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b5780 msgr2=0x7f9be40b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b5780 0x7f9be40b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 -- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9be40b6610 msgr2=0x7f9be41f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9be40b6610 0x7f9be41f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.518+0000 7f9be898b640 1 -- 172.21.15.146:0/3661090936 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be41eb870 con 0x7f9be415ffa0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.519+0000 7f9be898b640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 0x7f9be41f6090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00cad0 tx=0x7f9bdc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.519+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc017070 con 0x7f9be415ffa0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.519+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bdc00ac30 con 0x7f9be415ffa0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.519+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be41eb590 con 0x7f9be415ffa0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.519+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc012400 con 0x7f9be415ffa0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.519+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be4148d10 con 0x7f9be415ffa0 2024-04-25T15:20:18.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.520+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be41eb300 con 0x7f9be415ffa0 2024-04-25T15:20:18.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.520+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9bdc01a070 con 0x7f9be415ffa0 2024-04-25T15:20:18.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.521+0000 7f9be0ff9640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9bc40744b0 0x7f9bc40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.521+0000 7f9be3fff640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9bc40744b0 0x7f9bc40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.521+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9bdc095260 con 0x7f9be415ffa0 2024-04-25T15:20:18.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.522+0000 7f9be3fff640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9bc40744b0 0x7f9bc40769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4034000 tx=0x7f9bd402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.526+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9be40b5b90 con 0x7f9be415ffa0 2024-04-25T15:20:18.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.738+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9be406a0e0 con 0x7f9be415ffa0 2024-04-25T15:20:18.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.739+0000 7f9be0ff9640 1 -- 172.21.15.146:0/3661090936 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7f9bdc05f050 con 0x7f9be415ffa0 2024-04-25T15:20:18.741 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:18.741 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:18.741 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9bc40744b0 msgr2=0x7f9bc40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9bc40744b0 0x7f9bc40769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4034000 tx=0x7f9bd402e000 comp rx=0 tx=0).stop 2024-04-25T15:20:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 msgr2=0x7f9be41f6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 0x7f9be41f6090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00cad0 tx=0x7f9bdc00cf90 comp rx=0 tx=0).stop 2024-04-25T15:20:18.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 shutdown_connections 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9bc40744b0 0x7f9bc40769c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9be415ffa0 0x7f9be41f6090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9be40b6610 0x7f9be41f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 --2- 172.21.15.146:0/3661090936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9be40b5780 0x7f9be40b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 >> 172.21.15.146:0/3661090936 conn(0x7f9be40d0240 msgr2=0x7f9be40b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 shutdown_connections 2024-04-25T15:20:18.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.741+0000 7f9bea3ee640 1 -- 172.21.15.146:0/3661090936 wait complete. 2024-04-25T15:20:18.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:18.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.860+0000 7f04d1014640 1 Processor -- start 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.860+0000 7f04d1014640 1 -- start start 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04d1014640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 0x7f04cc16b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04d1014640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04cc0b8eb0 con 0x7f04cc06ee60 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04d1014640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04cc08ce00 con 0x7f04cc0b2ba0 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04d1014640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f04cc161840 con 0x7f04cc1691d0 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04cad9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f04cc06ee60 0x7f04cc163eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40738/0 (socket says 172.21.15.146:40738) 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04cad9d640 1 -- 172.21.15.146:0/4230903806 learned_addr learned my addr 172.21.15.146:0/4230903806 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04cb59e640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 0x7f04cc16b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373127487 0 0) 0x7f04cc0b8eb0 con 0x7f04cc06ee60 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04b4003520 con 0x7f04cc06ee60 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138025573 0 0) 0x7f04cc08ce00 con 0x7f04cc0b2ba0 2024-04-25T15:20:18.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04cc0b8eb0 con 0x7f04cc0b2ba0 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1791442263 0 0) 0x7f04b4003520 con 0x7f04cc06ee60 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 msgr2=0x7f04cc16b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 0x7f04cc16b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 >> v1:172.21.15.177:6789/0 conn(0x7f04cc0b2ba0 legacy=0x7f04cc1645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04cc08d0e0 con 0x7f04cc06ee60 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.862+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f04bc001b70 con 0x7f04cc06ee60 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 >> v1:172.21.15.146:6789/0 conn(0x7f04cc06ee60 legacy=0x7f04cc163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04b4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b4004800 0x7f04b4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 0x7f04b4005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc161840 con 0x7f04b4003b70 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04b400bb50 con 0x7f04b4004800 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f04b400bd20 con 0x7f04cc1691d0 2024-04-25T15:20:18.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cad9d640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04b4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04ca59c640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b4004800 0x7f04b4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cb59e640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 0x7f04b4005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cad9d640 1 -- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 msgr2=0x7f04b4005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cad9d640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc1691d0 0x7f04b4005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cad9d640 1 -- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b4004800 msgr2=0x7f04b4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cad9d640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b4004800 0x7f04b4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.863+0000 7f04cad9d640 1 -- 172.21.15.146:0/4230903806 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b4003520 con 0x7f04b4003b70 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04cad9d640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04b4003fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00c440 tx=0x7f04b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1309145226062ae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc032020 con 0x7f04b4003b70 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04bc005d40 con 0x7f04b4003b70 2024-04-25T15:20:18.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04c9d9b640 1 -- 172.21.15.146:0/4230903806 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00c0e0 con 0x7f04b4003b70 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04d1014640 1 -- 172.21.15.146:0/4230903806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 msgr2=0x7f04b4003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04d1014640 1 --2- 172.21.15.146:0/4230903806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04b4003fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00c440 tx=0x7f04b4005830 comp rx=0 tx=0).stop 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04cb59e640 1 -- 172.21.15.146:0/4230903806 reap_dead start 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04d1014640 1 -- 172.21.15.146:0/4230903806 shutdown_connections 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04d1014640 1 -- 172.21.15.146:0/4230903806 >> 172.21.15.146:0/4230903806 conn(0x7f04cc0b52e0 msgr2=0x7f04cc0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04d1014640 1 -- 172.21.15.146:0/4230903806 shutdown_connections 2024-04-25T15:20:18.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.864+0000 7f04d1014640 1 -- 172.21.15.146:0/4230903806 wait complete. 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.866+0000 7f04d1014640 1 Processor -- start 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.867+0000 7f04d1014640 1 -- start start 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04d1014640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04cc0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04d1014640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc0b7630 0x7f04cc15c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04d1014640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04cc1f6eb0 0x7f04cc1f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04d1014640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc06f440 con 0x7f04b4003b70 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04d1014640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc08d730 con 0x7f04cc1f6eb0 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04d1014640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f04cc08dbc0 con 0x7f04cc0b7630 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04cc0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04cc0b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60560/0 (socket says 172.21.15.146:60560) 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 -- 172.21.15.146:0/3095723285 learned_addr learned my addr 172.21.15.146:0/3095723285 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:18.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04ca59c640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc0b7630 0x7f04cc15c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cb59e640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04cc1f6eb0 0x7f04cc1f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 -- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc0b7630 msgr2=0x7f04cc15c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc0b7630 0x7f04cc15c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 -- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04cc1f6eb0 msgr2=0x7f04cc1f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04cc1f6eb0 0x7f04cc1f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 -- 172.21.15.146:0/3095723285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04cc1facf0 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.868+0000 7f04cad9d640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04cc0b04c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00c560 tx=0x7f04cc08cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.869+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc0012f0 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.869+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04bc007890 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.869+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04cc1fa450 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.869+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc043680 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.869+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04cc1faa10 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.870+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04cc069030 con 0x7f04b4003b70 2024-04-25T15:20:18.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.870+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f04bc035020 con 0x7f04b4003b70 2024-04-25T15:20:18.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.871+0000 7f04b37fe640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f04b80744b0 0x7f04b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:18.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.871+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f04bc0b61e0 con 0x7f04b4003b70 2024-04-25T15:20:18.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.871+0000 7f04ca59c640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f04b80744b0 0x7f04b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:18.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.871+0000 7f04ca59c640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f04b80744b0 0x7f04b80769c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0a0ee0 tx=0x7f04c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:18.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:18.875+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04cc069030 con 0x7f04b4003b70 2024-04-25T15:20:19.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.050+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04cc06a0e0 con 0x7f04b4003b70 2024-04-25T15:20:19.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.051+0000 7f04b37fe640 1 -- 172.21.15.146:0/3095723285 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18562 (secure 0 0 0) 0x7f04bc048060 con 0x7f04b4003b70 2024-04-25T15:20:19.053 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:19.053 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:19:52.485625+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:19.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f04b80744b0 msgr2=0x7f04b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f04b80744b0 0x7f04b80769c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0a0ee0 tx=0x7f04c0009040 comp rx=0 tx=0).stop 2024-04-25T15:20:19.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 msgr2=0x7f04cc0b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04cc0b04c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00c560 tx=0x7f04cc08cfd0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 shutdown_connections 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f04b80744b0 0x7f04b80769c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04cc1f6eb0 0x7f04cc1f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04cc0b7630 0x7f04cc15c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 --2- 172.21.15.146:0/3095723285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04b4003b70 0x7f04cc0b04c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 >> 172.21.15.146:0/3095723285 conn(0x7f04cc0b52e0 msgr2=0x7f04cc160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 shutdown_connections 2024-04-25T15:20:19.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.053+0000 7f04d1014640 1 -- 172.21.15.146:0/3095723285 wait complete. 2024-04-25T15:20:19.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.173+0000 7fa8941ce640 1 Processor -- start 2024-04-25T15:20:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.173+0000 7fa8941ce640 1 -- start start 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.174+0000 7fa8941ce640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa88c0cfbe0 0x7fa88c0cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa8941ce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c0cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa8941ce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88c0b4a60 con 0x7fa88c0d19d0 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa8941ce640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa88c0b5020 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa8941ce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa88c0b4d40 con 0x7fa88c0cce10 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa88c0cfbe0 0x7fa88c0cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa88c0cfbe0 0x7fa88c0cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45938/0 (socket says 172.21.15.146:45938) 2024-04-25T15:20:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 -- 172.21.15.146:0/98166228 learned_addr learned my addr 172.21.15.146:0/98166228 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891769640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fa88c0d19d0 0x7fa88c0d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:39088/0 (socket says 172.21.15.146:39088) 2024-04-25T15:20:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa89276b640 1 --2- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c0cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 -- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 msgr2=0x7fa88c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 --2- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c0cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 -- 172.21.15.146:0/98166228 >> v1:172.21.15.177:6789/0 conn(0x7fa88c0d19d0 legacy=0x7fa88c0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.175+0000 7fa891f6a640 1 -- 172.21.15.146:0/98166228 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88c0cead0 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.176+0000 7fa891f6a640 1 --2- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cf260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa888009ea0 tx=0x7fa88802a7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b0199c0802546ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.176+0000 7fa890f68640 1 -- 172.21.15.146:0/98166228 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa888036070 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.176+0000 7fa890f68640 1 -- 172.21.15.146:0/98166228 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa888031600 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.176+0000 7fa890f68640 1 -- 172.21.15.146:0/98166228 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8880319d0 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 -- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0cfbe0 msgr2=0x7fa88c0cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 --2- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cf260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa888009ea0 tx=0x7fa88802a7b0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 -- 172.21.15.146:0/98166228 shutdown_connections 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 --2- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c0cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 --2- 172.21.15.146:0/98166228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cf260 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 -- 172.21.15.146:0/98166228 >> 172.21.15.146:0/98166228 conn(0x7fa88c0b73b0 msgr2=0x7fa88c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 -- 172.21.15.146:0/98166228 shutdown_connections 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.177+0000 7fa8941ce640 1 -- 172.21.15.146:0/98166228 wait complete. 2024-04-25T15:20:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.179+0000 7fa8941ce640 1 Processor -- start 2024-04-25T15:20:19.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.179+0000 7fa8941ce640 1 -- start start 2024-04-25T15:20:19.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa8941ce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa8941ce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa8941ce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0d19d0 0x7fa88c1f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa8941ce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa88c0cf7a0 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa8941ce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa88c0cd760 con 0x7fa88c0d19d0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa8941ce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa88c1525b0 con 0x7fa88c0cce10 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa891769640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa891769640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60572/0 (socket says 172.21.15.146:60572) 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa891769640 1 -- 172.21.15.146:0/3395184043 learned_addr learned my addr 172.21.15.146:0/3395184043 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa89276b640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0d19d0 0x7fa88c1f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.180+0000 7fa891f6a640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa891769640 1 -- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 msgr2=0x7fa88c159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa891769640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa891769640 1 -- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0d19d0 msgr2=0x7fa88c1f94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa891769640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0d19d0 0x7fa88c1f94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa891769640 1 -- 172.21.15.146:0/3395184043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88c1fa250 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa891769640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cbec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00c440 tx=0x7fa87c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0074e0 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c007680 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88c150790 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c004a20 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.181+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa88c150d50 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.182+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa88c150580 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.183+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa87c01a020 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.183+0000 7fa8827fc640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa8700744b0 0x7fa8700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.183+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa87c094af0 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.183+0000 7fa891f6a640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa8700744b0 0x7fa8700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.184+0000 7fa891f6a640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa8700744b0 0x7fa8700769c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa888033000 tx=0x7fa88802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.188+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa87c05e980 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.399+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa88c0cf920 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.400+0000 7fa8827fc640 1 -- 172.21.15.146:0/3395184043 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7fa87c062c80 con 0x7fa88c0cfbe0 2024-04-25T15:20:19.402 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:19.402 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:19.402 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa8700744b0 msgr2=0x7fa8700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa8700744b0 0x7fa8700769c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa888033000 tx=0x7fa88802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 msgr2=0x7fa88c0cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cbec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00c440 tx=0x7fa87c00c900 comp rx=0 tx=0).stop 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 shutdown_connections 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa8700744b0 0x7fa8700769c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa88c0d19d0 0x7fa88c1f94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa88c0cfbe0 0x7fa88c0cbec0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 --2- 172.21.15.146:0/3395184043 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa88c0cce10 0x7fa88c159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 >> 172.21.15.146:0/3395184043 conn(0x7fa88c0b73b0 msgr2=0x7fa88c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 shutdown_connections 2024-04-25T15:20:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.402+0000 7fa8941ce640 1 -- 172.21.15.146:0/3395184043 wait complete. 2024-04-25T15:20:19.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:19:45.824153+0000', 'modified': '2024-04-25T15:20:07.899992+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6834/3903573056', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 3903573056}, {'type': 'v1', 'addr': '172.21.15.177:6834', 'nonce': 3903573056}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-04-25T15:20:19.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:20:19.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T15:20:19.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.929+0000 7f99ffa96640 1 Processor -- start 2024-04-25T15:20:19.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.929+0000 7f99ffa96640 1 -- start start 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99ffa96640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99f8162890 0x7f99f808d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99ffa96640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99f80b4df0 0x7f99f8161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99ffa96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99f8162520 con 0x7f99f8162110 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99ffa96640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f99f80d23a0 con 0x7f99f8162890 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99ffa96640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f99f80d1790 con 0x7f99f80b4df0 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd832640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99f8162890 0x7f99f808d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fe033640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f99f8162110 0x7f99f815e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:40752/0 (socket says 172.21.15.146:40752) 2024-04-25T15:20:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fe033640 1 -- 172.21.15.146:0/3938213469 learned_addr learned my addr 172.21.15.146:0/3938213469 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd832640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99f8162890 0x7f99f808d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60576/0 (socket says 172.21.15.146:60576) 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd031640 1 --2- 172.21.15.146:0/3938213469 >> v2:172.21.15.177:3300/0 conn(0x7f99f80b4df0 0x7f99f8161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd832640 1 -- 172.21.15.146:0/3938213469 >> v2:172.21.15.177:3300/0 conn(0x7f99f80b4df0 msgr2=0x7f99f8161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd832640 1 --2- 172.21.15.146:0/3938213469 >> v2:172.21.15.177:3300/0 conn(0x7f99f80b4df0 0x7f99f8161bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd832640 1 -- 172.21.15.146:0/3938213469 >> v1:172.21.15.146:6789/0 conn(0x7f99f8162110 legacy=0x7f99f815e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fc830640 1 -- 172.21.15.146:0/3938213469 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605153124 0 0) 0x7f99f0001110 con 0x7f99f8162110 2024-04-25T15:20:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.931+0000 7f99fd832640 1 -- 172.21.15.146:0/3938213469 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f80d1a70 con 0x7f99f8162890 2024-04-25T15:20:19.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.932+0000 7f99fd832640 1 --2- 172.21.15.146:0/3938213469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f808d670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f99f80d1370 tx=0x7f99f402a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c71dcc25ceba67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.932+0000 7f99fc830640 1 -- 172.21.15.146:0/3938213469 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4007740 con 0x7f99f8162890 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.932+0000 7f99fc830640 1 -- 172.21.15.146:0/3938213469 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f402ae50 con 0x7f99f8162890 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.932+0000 7f99fc830640 1 -- 172.21.15.146:0/3938213469 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4007b00 con 0x7f99f8162890 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 -- 172.21.15.146:0/3938213469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 msgr2=0x7f99f808d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 --2- 172.21.15.146:0/3938213469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f808d670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f99f80d1370 tx=0x7f99f402a630 comp rx=0 tx=0).stop 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 -- 172.21.15.146:0/3938213469 shutdown_connections 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 --2- 172.21.15.146:0/3938213469 >> v2:172.21.15.177:3300/0 conn(0x7f99f80b4df0 0x7f99f8161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 --2- 172.21.15.146:0/3938213469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f808d670 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 -- 172.21.15.146:0/3938213469 >> 172.21.15.146:0/3938213469 conn(0x7f99f80b52e0 msgr2=0x7f99f80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 -- 172.21.15.146:0/3938213469 shutdown_connections 2024-04-25T15:20:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.933+0000 7f99ffa96640 1 -- 172.21.15.146:0/3938213469 wait complete. 2024-04-25T15:20:19.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.935+0000 7f99ffa96640 1 Processor -- start 2024-04-25T15:20:19.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.935+0000 7f99ffa96640 1 -- start start 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99ffa96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f80b4df0 0x7f99f814e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99ffa96640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99f8162110 0x7f99f80a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99ffa96640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99ffa96640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f99f80d2830 con 0x7f99f8162890 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99ffa96640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99f80d2030 con 0x7f99f80b4df0 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99ffa96640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f99f808dbb0 con 0x7f99f8162110 2024-04-25T15:20:19.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f80b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60578/0 (socket says 172.21.15.146:60578) 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 -- 172.21.15.146:0/487515232 learned_addr learned my addr 172.21.15.146:0/487515232 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fd832640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f80b4df0 0x7f99f814e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fd031640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99f8162110 0x7f99f80a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 -- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99f8162110 msgr2=0x7f99f80a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99f8162110 0x7f99f80a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 -- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f80b4df0 msgr2=0x7f99f814e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f80b4df0 0x7f99f814e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.936+0000 7f99fe033640 1 -- 172.21.15.146:0/487515232 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f80a13f0 con 0x7f99f8162890 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.937+0000 7f99fe033640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f80b2170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c3e0 tx=0x7f99f000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.937+0000 7f99ee7fc640 1 -- 172.21.15.146:0/487515232 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00074e0 con 0x7f99f8162890 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.937+0000 7f99ee7fc640 1 -- 172.21.15.146:0/487515232 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f00047e0 con 0x7f99f8162890 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.937+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f814be10 con 0x7f99f8162890 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.937+0000 7f99ee7fc640 1 -- 172.21.15.146:0/487515232 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0004bd0 con 0x7f99f8162890 2024-04-25T15:20:19.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.937+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f814c3d0 con 0x7f99f8162890 2024-04-25T15:20:19.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.939+0000 7f99ee7fc640 1 -- 172.21.15.146:0/487515232 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f99f0011070 con 0x7f99f8162890 2024-04-25T15:20:19.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.939+0000 7f99ee7fc640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99dc0744b0 0x7f99dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.939+0000 7f99ee7fc640 1 -- 172.21.15.146:0/487515232 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f99f00950d0 con 0x7f99f8162890 2024-04-25T15:20:19.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.939+0000 7f99fd832640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99dc0744b0 0x7f99dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.939+0000 7f99fd832640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99dc0744b0 0x7f99dc0769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f99f4034000 tx=0x7f99f4033040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.940+0000 7f9a06a25740 1 Processor -- start 2024-04-25T15:20:19.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.940+0000 7f9a06a25740 1 -- start start 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e719cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719a5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f9a06a25740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e71909740 con 0x564e715e2590 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e718a53a0 con 0x564e71603270 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x564e71905700 con 0x564e715da3b0 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60592/0 (socket says 172.21.15.146:60592) 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.941+0000 7f99fe033640 1 -- 172.21.15.146:0/2207076598 learned_addr learned my addr 172.21.15.146:0/2207076598 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fd832640 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fd031640 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e719cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fe033640 1 -- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 msgr2=0x564e719cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fe033640 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e719cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fe033640 1 -- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 msgr2=0x564e719cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fe033640 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fe033640 1 -- 172.21.15.146:0/2207076598 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e718c1750 con 0x564e715e2590 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99fe033640 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719a5260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f99f005b410 tx=0x7f99f005b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c89247acd509003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99db7fe640 1 -- 172.21.15.146:0/2207076598 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f005ee20 con 0x564e715e2590 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99db7fe640 1 -- 172.21.15.146:0/2207076598 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f00631d0 con 0x564e715e2590 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.942+0000 7f99db7fe640 1 -- 172.21.15.146:0/2207076598 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f005e2c0 con 0x564e715e2590 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 -- 172.21.15.146:0/2207076598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 msgr2=0x564e719a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719a5260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f99f005b410 tx=0x7f99f005b470 comp rx=0 tx=0).stop 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 -- 172.21.15.146:0/2207076598 shutdown_connections 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719a5260 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e719cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2207076598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 -- 172.21.15.146:0/2207076598 >> 172.21.15.146:0/2207076598 conn(0x564e715f73c0 msgr2=0x564e719a49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 -- 172.21.15.146:0/2207076598 shutdown_connections 2024-04-25T15:20:19.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.943+0000 7f9a06a25740 1 -- 172.21.15.146:0/2207076598 wait complete. 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.944+0000 7f9a06a25740 1 -- start start 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.944+0000 7f9a06a25740 1 Processor -- start 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e719d0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e719f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e719f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e71909740 con 0x564e71603270 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e71890fe0 con 0x564e715da3b0 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x564e719d3560 con 0x564e715e2590 2024-04-25T15:20:19.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f9a06a25740 10 client.0 _ll_register_callbacks cb 0x564e718a7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e719f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e719f94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60594/0 (socket says 172.21.15.146:60594) 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f99fe033640 1 -- 172.21.15.146:0/2925803280 learned_addr learned my addr 172.21.15.146:0/2925803280 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f99fd031640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e719f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.945+0000 7f99fd832640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e719d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99fe033640 1 -- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 msgr2=0x564e719f8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99fe033640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e719f8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99fe033640 1 -- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 msgr2=0x564e719d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99fe033640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e719d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99fe033640 1 -- 172.21.15.146:0/2925803280 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e7161b2d0 con 0x564e71603270 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99fe033640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e719f94c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f99f005edb0 tx=0x7f99f0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de39f791e69b31f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99d8ff9640 1 -- 172.21.15.146:0/2925803280 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0067c80 con 0x564e71603270 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99d8ff9640 1 -- 172.21.15.146:0/2925803280 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f99f005daf0 con 0x564e71603270 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99f00631d0 con 0x564e71603270 2024-04-25T15:20:19.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99d8ff9640 1 -- 172.21.15.146:0/2925803280 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f99f005ee20 con 0x564e71603270 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 10 client.4866 fetch_fsmap learned FSMap version 8 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x564e716e56b0 con 0x564e71603270 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99d8ff9640 1 -- 172.21.15.146:0/2925803280 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f00631d0 con 0x564e71603270 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f99d8ff9640 1 -- 172.21.15.146:0/2925803280 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f99f005dd10 con 0x564e71603270 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 10 client.4866 fetch_fsmap finished waiting for FSMap version 8 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 10 client.4866 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 20 client.4866 populate_metadata read hostname 'smithi146' 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e715c1e40 0x564e719d29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 4 client.4866 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.946+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 --> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e71917350 con 0x564e715c1e40 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.947+0000 7f99fd832640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e715c1e40 0x564e719d29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.947+0000 7f99d8ff9640 10 client.4866 ms_handle_connect on v2:172.21.15.177:6832/3903573056 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.947+0000 7f99fd832640 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e715c1e40 0x564e719d29b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a3a110fcb847b2e server_cookie=d423e1c3aee9758 in_seq=0 out_seq=0 2024-04-25T15:20:19.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f99d8ff9640 1 -- 172.21.15.146:0/2925803280 <== mds.0 v2:172.21.15.177:6832/3903573056 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564e71917350 con 0x564e715c1e40 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f99d8ff9640 10 client.4866 handle_command_reply: tid=0 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 client.4866 shutdown 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 msgr2=0x564e719f94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e719f94c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f99f005edb0 tx=0x7f99f0000ed0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 shutdown_connections 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e715c1e40 0x564e719d29b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e719f94c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e719f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f99d8ff9640 0 client.4866 ms_handle_reset on v2:172.21.15.177:6832/3903573056 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 --2- 172.21.15.146:0/2925803280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e719d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 >> 172.21.15.146:0/2925803280 conn(0x564e715f73c0 msgr2=0x564e719a49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 shutdown_connections 2024-04-25T15:20:19.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 1 -- 172.21.15.146:0/2925803280 wait complete. 2024-04-25T15:20:19.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.948+0000 7f9a06a25740 20 client.4866 trim_cache size 0 max 16384 2024-04-25T15:20:19.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.967+0000 7f9a06a25740 1 Processor -- start 2024-04-25T15:20:19.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.967+0000 7f9a06a25740 1 -- start start 2024-04-25T15:20:19.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e71a5dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e7199e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e7199e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f9a06a25740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f99f8200e20 con 0x564e71603270 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99f809e390 con 0x564e715da3b0 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f99f82007d0 con 0x564e715e2590 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e7199e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f99fe033640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e7199e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60606/0 (socket says 172.21.15.146:60606) 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f99fe033640 1 -- 172.21.15.146:0/652527023 learned_addr learned my addr 172.21.15.146:0/652527023 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f99fe033640 1 -- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 msgr2=0x564e7199e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.968+0000 7f99fe033640 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e7199e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99fe033640 1 -- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 msgr2=0x564e71a5dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99fd832640 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e71a5dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99fe033640 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e71a5dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99fe033640 1 -- 172.21.15.146:0/652527023 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e719cf210 con 0x564e71603270 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99fe033640 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e7199e840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f99f0006010 tx=0x7f99f008e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7417b9e1b4d1ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99daffd640 1 -- 172.21.15.146:0/652527023 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00631d0 con 0x564e71603270 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99daffd640 1 -- 172.21.15.146:0/652527023 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0084070 con 0x564e71603270 2024-04-25T15:20:19.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f99daffd640 1 -- 172.21.15.146:0/652527023 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0067c80 con 0x564e71603270 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f9a06a25740 1 -- 172.21.15.146:0/652527023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 msgr2=0x564e7199e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.969+0000 7f9a06a25740 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e7199e840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f99f0006010 tx=0x7f99f008e040 comp rx=0 tx=0).stop 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 -- 172.21.15.146:0/652527023 shutdown_connections 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e71603270 0x564e7199e840 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715e2590 0x564e7199e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 --2- 172.21.15.146:0/652527023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e715da3b0 0x564e71a5dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 -- 172.21.15.146:0/652527023 >> 172.21.15.146:0/652527023 conn(0x564e715f73c0 msgr2=0x564e719ea4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 -- 172.21.15.146:0/652527023 shutdown_connections 2024-04-25T15:20:19.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 -- 172.21.15.146:0/652527023 wait complete. 2024-04-25T15:20:19.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 -- start start 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.970+0000 7f9a06a25740 1 Processor -- start 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e71a60d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719e9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719ea530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e71909740 con 0x564e715e2590 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e71890fe0 con 0x564e71603270 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x564e719fc940 con 0x564e715da3b0 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f9a06a25740 10 client.0 _ll_register_callbacks cb 0x7f99f005ec80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719e9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719e9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60618/0 (socket says 172.21.15.146:60618) 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 -- 172.21.15.146:0/307567585 learned_addr learned my addr 172.21.15.146:0/307567585 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:19.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fe033640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719ea530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd832640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e71a60d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 -- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 msgr2=0x564e71a60d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e71a60d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 -- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 msgr2=0x564e719ea530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719ea530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd031640 1 -- 172.21.15.146:0/307567585 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e719edec0 con 0x564e715e2590 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fd832640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e71a60d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.972+0000 7f99fe033640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719ea530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99fd031640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719e9ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x564e719a6e80 tx=0x7f99e80129e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e5d0cdfa4d8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99d8ff9640 1 -- 172.21.15.146:0/307567585 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e801b020 con 0x564e715e2590 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99d8ff9640 1 -- 172.21.15.146:0/307567585 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f99e8004240 con 0x564e715e2590 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99f0084070 con 0x564e715e2590 2024-04-25T15:20:19.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99d8ff9640 1 -- 172.21.15.146:0/307567585 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f99e8025050 con 0x564e715e2590 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 10 client.4872 fetch_fsmap learned FSMap version 8 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x564e71a4c1f0 con 0x564e715e2590 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99d8ff9640 1 -- 172.21.15.146:0/307567585 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0084070 con 0x564e715e2590 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99d8ff9640 1 -- 172.21.15.146:0/307567585 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f99e8028020 con 0x564e715e2590 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 10 client.4872 fetch_fsmap finished waiting for FSMap version 8 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 10 client.4872 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 20 client.4872 populate_metadata read hostname 'smithi146' 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e71a0bda0 0x564e71a0c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 4 client.4872 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 --> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] -- command(tid 0: {"prefix": "status"}) -- 0x564e719d1b60 con 0x564e71a0bda0 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.973+0000 7f99fd832640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e71a0bda0 0x564e71a0c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f99d8ff9640 10 client.4872 ms_handle_connect on v2:172.21.15.177:6832/3903573056 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f99fd832640 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e71a0bda0 0x564e71a0c1b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f54c3c306ada0545 server_cookie=df4601535ced9226 in_seq=0 out_seq=0 2024-04-25T15:20:19.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f99d8ff9640 1 -- 172.21.15.146:0/307567585 <== mds.0 v2:172.21.15.177:6832/3903573056 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x564e719d1b60 con 0x564e71a0bda0 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f99d8ff9640 10 client.4872 handle_command_reply: tid=0 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 client.4872 shutdown 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 msgr2=0x564e719e9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719e9ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x564e719a6e80 tx=0x7f99e80129e0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 shutdown_connections 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:6832/3903573056,v1:172.21.15.177:6834/3903573056] conn(0x564e71a0bda0 0x564e71a0c1b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e71603270 0x564e719ea530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f99d8ff9640 0 client.4872 ms_handle_reset on v2:172.21.15.177:6832/3903573056 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e715e2590 0x564e719e9ff0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 --2- 172.21.15.146:0/307567585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564e715da3b0 0x564e71a60d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 >> 172.21.15.146:0/307567585 conn(0x564e715f73c0 msgr2=0x564e719cca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 shutdown_connections 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 1 -- 172.21.15.146:0/307567585 wait complete. 2024-04-25T15:20:19.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.974+0000 7f9a06a25740 20 client.4872 trim_cache size 0 max 16384 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "c6e9a678-3722-48f4-a297-70f509c13e29", 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4294, 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 33.514351197000003, 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 8, 2024-04-25T15:20:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 25, 2024-04-25T15:20:19.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 25, 2024-04-25T15:20:19.983 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 40.240929612000002 2024-04-25T15:20:19.983 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99dc0744b0 msgr2=0x7f99dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99dc0744b0 0x7f99dc0769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f99f4034000 tx=0x7f99f4033040 comp rx=0 tx=0).stop 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 msgr2=0x7f99f80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f80b2170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c3e0 tx=0x7f99f000c8a0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 shutdown_connections 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99dc0744b0 0x7f99dc0769c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f8162890 0x7f99f80b2170 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99f8162110 0x7f99f80a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 --2- 172.21.15.146:0/487515232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f80b4df0 0x7f99f814e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 >> 172.21.15.146:0/487515232 conn(0x7f99f80b52e0 msgr2=0x7f99f8072a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 shutdown_connections 2024-04-25T15:20:19.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:19.975+0000 7f99ffa96640 1 -- 172.21.15.146:0/487515232 wait complete. 2024-04-25T15:20:19.988 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T15:20:19.998 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T15:20:19.998 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T15:20:19.999 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T15:20:19.999 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T15:20:20.013 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T15:20:20.013 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-25T15:20:20.013 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-25T15:20:20.013 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:20:20.013 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:20.013 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-25T15:20:20.013 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:20:20.013 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:20.014 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-25T15:20:20.014 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:20:20.014 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:20.014 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-25T15:20:20.014 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:20:20.014 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:20.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.014 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:20:20.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.033 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link delete ceph-brx 2024-04-25T15:20:20.104 INFO:teuthology.orchestra.run.smithi177.stderr:Cannot find device "ceph-brx" 2024-04-25T15:20:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:20:20.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.105 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:20:20.121 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T15:20:20.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T15:20:20.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-04-25T15:20:20.283 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T15:20:20.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:20:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.392+0000 7fc6a55c7640 1 Processor -- start 2024-04-25T15:20:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.392+0000 7fc6a55c7640 1 -- start start 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc6a55c7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a00b8dc0 0x7fc6a0071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc6a55c7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6a0161ab0 0x7fc6a0161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc6a55c7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6a00cfb60 con 0x7fc6a00cf040 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc6a55c7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a00d2a20 con 0x7fc6a0161ab0 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc6a55c7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a00d2460 con 0x7fc6a00b8dc0 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6a0161ab0 0x7fc6a0161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6a0161ab0 0x7fc6a0161ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60620/0 (socket says 172.21.15.146:60620) 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 -- 172.21.15.146:0/4160559854 learned_addr learned my addr 172.21.15.146:0/4160559854 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69effd640 1 --2- 172.21.15.146:0/4160559854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a00b8dc0 0x7fc6a0071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 -- 172.21.15.146:0/4160559854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a00b8dc0 msgr2=0x7fc6a0071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 --2- 172.21.15.146:0/4160559854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a00b8dc0 0x7fc6a0071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 -- 172.21.15.146:0/4160559854 >> v1:172.21.15.177:6789/0 conn(0x7fc6a00cf040 legacy=0x7fc6a00cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.393+0000 7fc69f7fe640 1 -- 172.21.15.146:0/4160559854 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a00d2740 con 0x7fc6a0161ab0 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.394+0000 7fc69f7fe640 1 --2- 172.21.15.146:0/4160559854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a0161ab0 0x7fc6a0161ec0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6980098a0 tx=0x7fc69802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ae9cdf12c2d2c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.394+0000 7fc69dffb640 1 -- 172.21.15.146:0/4160559854 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698037070 con 0x7fc6a0161ab0 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.394+0000 7fc69dffb640 1 -- 172.21.15.146:0/4160559854 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc698032440 con 0x7fc6a0161ab0 2024-04-25T15:20:20.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.394+0000 7fc69dffb640 1 -- 172.21.15.146:0/4160559854 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698032810 con 0x7fc6a0161ab0 2024-04-25T15:20:20.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 -- 172.21.15.146:0/4160559854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a0161ab0 msgr2=0x7fc6a0161ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:20.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/4160559854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a0161ab0 0x7fc6a0161ec0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6980098a0 tx=0x7fc69802a200 comp rx=0 tx=0).stop 2024-04-25T15:20:20.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 -- 172.21.15.146:0/4160559854 shutdown_connections 2024-04-25T15:20:20.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/4160559854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a0161ab0 0x7fc6a0161ec0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/4160559854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a00b8dc0 0x7fc6a0071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 -- 172.21.15.146:0/4160559854 >> 172.21.15.146:0/4160559854 conn(0x7fc6a00b5360 msgr2=0x7fc6a00b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:20.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 -- 172.21.15.146:0/4160559854 shutdown_connections 2024-04-25T15:20:20.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.395+0000 7fc6a55c7640 1 -- 172.21.15.146:0/4160559854 wait complete. 2024-04-25T15:20:20.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.397+0000 7fc6a55c7640 1 Processor -- start 2024-04-25T15:20:20.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.397+0000 7fc6a55c7640 1 -- start start 2024-04-25T15:20:20.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc6a55c7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6a00b8dc0 0x7fc6a00cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:20.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc6a55c7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 0x7fc6a014bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:20.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc6a55c7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a0161ab0 0x7fc6a00721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:20.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc6a55c7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a0071b60 con 0x7fc6a00cf040 2024-04-25T15:20:20.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc6a55c7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a00ce4f0 con 0x7fc6a00b8dc0 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc6a55c7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a00d20f0 con 0x7fc6a0161ab0 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc69e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 0x7fc6a014bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc69e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 0x7fc6a014bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60632/0 (socket says 172.21.15.146:60632) 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.398+0000 7fc69e7fc640 1 -- 172.21.15.146:0/3707790371 learned_addr learned my addr 172.21.15.146:0/3707790371 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69f7fe640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a0161ab0 0x7fc6a00721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69effd640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6a00b8dc0 0x7fc6a00cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69e7fc640 1 -- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a0161ab0 msgr2=0x7fc6a00721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69e7fc640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a0161ab0 0x7fc6a00721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69e7fc640 1 -- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6a00b8dc0 msgr2=0x7fc6a00cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69e7fc640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6a00b8dc0 0x7fc6a00cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69e7fc640 1 -- 172.21.15.146:0/3707790371 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a00cadb0 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc69e7fc640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 0x7fc6a014bd00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc69400c3c0 tx=0x7fc69400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694017070 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6940052d0 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a00cb090 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694003830 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.399+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a00cb650 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.400+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a006a0e0 con 0x7fc6a00cf040 2024-04-25T15:20:20.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.401+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc69401a070 con 0x7fc6a00cf040 2024-04-25T15:20:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.401+0000 7fc6877fe640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc68c0744b0 0x7fc68c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.401+0000 7fc69effd640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc68c0744b0 0x7fc68c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.401+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc694094660 con 0x7fc6a00cf040 2024-04-25T15:20:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.402+0000 7fc69effd640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc68c0744b0 0x7fc68c0769c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00d3020 tx=0x7fc690009880 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:20.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.406+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc69405e4f0 con 0x7fc6a00cf040 2024-04-25T15:20:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.622+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc6a00b5f50 con 0x7fc6a00cf040 2024-04-25T15:20:20.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.622+0000 7fc6877fe640 1 -- 172.21.15.146:0/3707790371 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fc6940627f0 con 0x7fc6a00cf040 2024-04-25T15:20:20.624 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T15:20:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc68c0744b0 msgr2=0x7fc68c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc68c0744b0 0x7fc68c0769c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00d3020 tx=0x7fc690009880 comp rx=0 tx=0).stop 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 msgr2=0x7fc6a014bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 0x7fc6a014bd00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc69400c3c0 tx=0x7fc69400c880 comp rx=0 tx=0).stop 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 shutdown_connections 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc68c0744b0 0x7fc68c0769c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6a0161ab0 0x7fc6a00721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a00cf040 0x7fc6a014bd00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 --2- 172.21.15.146:0/3707790371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6a00b8dc0 0x7fc6a00cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 >> 172.21.15.146:0/3707790371 conn(0x7fc6a00b5360 msgr2=0x7fc6a00cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 shutdown_connections 2024-04-25T15:20:20.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20.624+0000 7fc6a55c7640 1 -- 172.21.15.146:0/3707790371 wait complete. 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:20:20.638 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:20:20.639 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:20:20.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.639 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:20:20.654 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:20.654 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:20.654 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:20.654 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42741sec preferred_lft 42741sec 2024-04-25T15:20:20.655 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:20:20.656 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:20.656 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T15:20:20.656 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:20.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:20.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:20.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.791 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:20:20.813 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:20:20.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.815 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:20:20.870 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:20:20.870 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:20:20.870 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:20:20.870 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:20:20.870 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:20:20.871 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:20.871 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:20.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:20:20.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:20:20.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:20:20.871 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:20.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:20.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:20.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:20.994 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:20:21.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:21.010 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-04-25T15:20:21.065 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:21.065 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:21.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:21.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T15:20:21.065 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:21.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:21.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:21.180 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T15:20:21.180 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:21.180 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:21.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T15:20:21.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T15:20:21.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T15:20:21.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T15:20:21.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T15:20:21.181 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:21.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:21.290 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:21.290 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:21.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-25T15:20:21.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T15:20:21.290 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:21.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:21.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:21.366 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T15:20:21.366 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:21.367 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:21.382 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:20:21.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:21.382 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:21.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:20:21.502 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:21.557 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:20:21.557 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:20:21.557 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:20:21.557 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:20:21.557 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:20:21.557 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:20:21.558 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:21.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:21.560 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:20:21.612 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:20:21.612 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-25T15:20:21.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:20:21.678 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:21.736 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:20:21.736 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:20:21.736 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:20:21.736 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:20:21.736 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:20:21.736 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:20:21.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:20:21.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:20:21.739 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:20:21.739 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:21.739 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:21.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:21.739 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:20:21.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:2024-04-25T15:20:21.742+0000 7f4984f21480 -1 init, newargv = 0x7f4974001210 newargc=15 2024-04-25T15:20:21.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[45337]: starting ceph client 2024-04-25T15:20:21.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[45337]: starting fuse 2024-04-25T15:20:21.758 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:20:21.759 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:20:21.759 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:21.760 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.0")) 2024-04-25T15:20:21.761 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:20:21.876 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.0.45337.asok 2024-04-25T15:20:21.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:21.887 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45337.asok 2024-04-25T15:20:21.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:21.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45337.asok status 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "0", 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:20:22.030 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "45337", 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4887, 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:20:22.031 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4887 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154176697 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:22.032 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154176697 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4887 192.168.0.1:0/154176697", 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/154176697", 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 8, 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 25, 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:20:22.033 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:20:22.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:20:22.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.145+0000 7f1378630640 1 Processor -- start 2024-04-25T15:20:22.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.145+0000 7f1378630640 1 -- start start 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1378630640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1378630640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13700d19d0 0x7f13700d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1378630640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13700b4a60 con 0x7f13700b3920 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1378630640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f13700b5020 con 0x7f13700d19d0 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1378630640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f13700b4d40 con 0x7f13700cfbe0 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13700d19d0 0x7f13700d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:22.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13700d19d0 0x7f13700d1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35294/0 (socket says 172.21.15.146:35294) 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 -- 172.21.15.146:0/2188474505 learned_addr learned my addr 172.21.15.146:0/2188474505 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f13763cc640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 -- 172.21.15.146:0/2188474505 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 msgr2=0x7f13700cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 -- 172.21.15.146:0/2188474505 >> v1:172.21.15.177:6789/0 conn(0x7f13700b3920 legacy=0x7f13700b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f1375bcb640 1 -- 172.21.15.146:0/2188474505 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13700cead0 con 0x7f13700d19d0 2024-04-25T15:20:22.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.147+0000 7f13763cc640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:22.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.148+0000 7f1375bcb640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f13700d1de0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1360009870 tx=0x7f136002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52972b2b3afd9d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:22.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.148+0000 7f13753ca640 1 -- 172.21.15.146:0/2188474505 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360037070 con 0x7f13700d19d0 2024-04-25T15:20:22.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.148+0000 7f13753ca640 1 -- 172.21.15.146:0/2188474505 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f136002ae50 con 0x7f13700d19d0 2024-04-25T15:20:22.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.148+0000 7f13753ca640 1 -- 172.21.15.146:0/2188474505 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360032860 con 0x7f13700d19d0 2024-04-25T15:20:22.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 -- 172.21.15.146:0/2188474505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 msgr2=0x7f13700d1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f13700d1de0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1360009870 tx=0x7f136002a300 comp rx=0 tx=0).stop 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 -- 172.21.15.146:0/2188474505 shutdown_connections 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f13700d1de0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 --2- 172.21.15.146:0/2188474505 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 -- 172.21.15.146:0/2188474505 >> 172.21.15.146:0/2188474505 conn(0x7f13700b73b0 msgr2=0x7f13700b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 -- 172.21.15.146:0/2188474505 shutdown_connections 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.149+0000 7f1378630640 1 -- 172.21.15.146:0/2188474505 wait complete. 2024-04-25T15:20:22.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.151+0000 7f1378630640 1 Processor -- start 2024-04-25T15:20:22.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.151+0000 7f1378630640 1 -- start start 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1378630640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700b3920 0x7f13700cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1378630640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1378630640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f137014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1378630640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f13700cf7a0 con 0x7f13700d19d0 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1378630640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13700d2320 con 0x7f13700b3920 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1378630640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f13700c8f80 con 0x7f13700cfbe0 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f137014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f137014f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35300/0 (socket says 172.21.15.146:35300) 2024-04-25T15:20:22.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 -- 172.21.15.146:0/3968855035 learned_addr learned my addr 172.21.15.146:0/3968855035 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f13763cc640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700b3920 0x7f13700cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 -- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 msgr2=0x7f13700cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 -- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700b3920 msgr2=0x7f13700cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700b3920 0x7f13700cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 -- 172.21.15.146:0/3968855035 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1370156500 con 0x7f13700d19d0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f13763cc640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700b3920 0x7f13700cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1376bcd640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f137014f240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f136800c390 tx=0x7f136800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.152+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1368017070 con 0x7f13700d19d0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.153+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13680052d0 con 0x7f13700d19d0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.153+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1368003810 con 0x7f13700d19d0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.153+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13700c7b20 con 0x7f13700d19d0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.153+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13700c7040 con 0x7f13700d19d0 2024-04-25T15:20:22.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.153+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137006a0e0 con 0x7f13700d19d0 2024-04-25T15:20:22.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.154+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f136801a070 con 0x7f13700d19d0 2024-04-25T15:20:22.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.155+0000 7f1366ffd640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13540744e0 0x7f13540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:22.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.155+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1368094870 con 0x7f13700d19d0 2024-04-25T15:20:22.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.155+0000 7f13763cc640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13540744e0 0x7f13540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:22.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.155+0000 7f13763cc640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13540744e0 0x7f13540769f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f136c009540 tx=0x7f136c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:22.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.160+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1370156500 con 0x7f13700d19d0 2024-04-25T15:20:22.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.374+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f13700cf920 con 0x7f13700d19d0 2024-04-25T15:20:22.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.374+0000 7f1366ffd640 1 -- 172.21.15.146:0/3968855035 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f136805ecc0 con 0x7f13700d19d0 2024-04-25T15:20:22.376 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T15:20:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13540744e0 msgr2=0x7f13540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13540744e0 0x7f13540769f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f136c009540 tx=0x7f136c00b040 comp rx=0 tx=0).stop 2024-04-25T15:20:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 msgr2=0x7f137014f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f137014f240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f136800c390 tx=0x7f136800c850 comp rx=0 tx=0).stop 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 shutdown_connections 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13540744e0 0x7f13540769f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13700d19d0 0x7f137014f240 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f13700cfbe0 0x7f13700cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 --2- 172.21.15.146:0/3968855035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700b3920 0x7f13700cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 >> 172.21.15.146:0/3968855035 conn(0x7f13700b73b0 msgr2=0x7f13701e5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 shutdown_connections 2024-04-25T15:20:22.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22.376+0000 7f1378630640 1 -- 172.21.15.146:0/3968855035 wait complete. 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:20:22.389 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:20:22.390 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:20:22.390 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:20:22.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.390 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:20:22.405 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:20:22.406 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42739sec preferred_lft 42739sec 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link tentative 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:20:22.407 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:22.408 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link tentative 2024-04-25T15:20:22.408 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.408 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:20:22.468 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:20:22.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.469 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:20:22.526 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:20:22.526 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:20:22.526 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:20:22.526 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:20:22.526 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:20:22.527 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:22.527 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:22.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:20:22.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:20:22.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:20:22.527 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:22.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:22.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:22.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.651 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:20:22.666 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:20:22.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.667 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-04-25T15:20:22.721 INFO:teuthology.orchestra.run.smithi146.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-25T15:20:22.722 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:22.722 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:22.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:20:22.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-25T15:20:22.722 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:22.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:22.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:22.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.836 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T15:20:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether c2:42:fd:0a:cb:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::c042:fdff:fe0a:cb6d/64 scope link tentative 2024-04-25T15:20:22.860 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:22.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T15:20:22.925 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:22.925 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.926 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:20:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether c2:42:fd:0a:cb:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:20:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:20:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::c042:fdff:fe0a:cb6d/64 scope link 2024-04-25T15:20:22.926 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:22.927 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-25T15:20:22.927 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:22.927 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:22.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-25T15:20:22.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T15:20:22.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-25T15:20:22.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-25T15:20:22.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-25T15:20:22.928 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:23.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:23.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:23.076 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:23.076 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:20:23.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.1 up 2024-04-25T15:20:23.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T15:20:23.076 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:20:23.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:23.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:23.152 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-25T15:20:23.152 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:23.153 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:23.167 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:20:23.167 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:23.167 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:23.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:20:23.285 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:23.341 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:20:23.341 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:20:23.341 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:20:23.341 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:20:23.341 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:20:23.342 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:20:23.343 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:20:23.344 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:23.344 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:23.344 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:23.344 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:20:23.344 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:20:23.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:23.344 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:20:23.396 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:20:23.396 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-25T15:20:23.397 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-25T15:20:23.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:20:23.463 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:23.521 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:20:23.522 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:23.523 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:20:23.524 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:23.524 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:23.524 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:23.524 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:23.524 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:23.524 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:20:23.525 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:23.526 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:20:23.526 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:20:23.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:23.526 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:20:23.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:2024-04-25T15:20:23.526+0000 7f4bd8b29480 -1 init, newargv = 0x7f4bbc053070 newargc=15 2024-04-25T15:20:23.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[45835]: starting ceph client 2024-04-25T15:20:23.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[45835]: starting fuse 2024-04-25T15:20:23.543 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:20:23.543 INFO:teuthology.orchestra.run.smithi146.stdout:44 2024-04-25T15:20:23.544 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:20:23.544 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:20:23.545 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:20:23.546 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.2")) 2024-04-25T15:20:23.546 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:20:23.663 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.2.45835.asok 2024-04-25T15:20:23.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:23.674 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45835.asok 2024-04-25T15:20:23.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:23.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45835.asok status 2024-04-25T15:20:23.818 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:20:23.818 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:20:23.818 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:20:23.818 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:20:23.818 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "2", 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "45835", 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4902, 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4902 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:23.819 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3633205019 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3633205019 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:20:23.820 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4902 192.168.0.2:0/3633205019", 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.2:0/3633205019", 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 8, 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 25, 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:20:23.821 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:20:23.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:20:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.931+0000 7f8f3f15e640 1 Processor -- start 2024-04-25T15:20:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.931+0000 7f8f3f15e640 1 -- start start 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3f15e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 0x7f8f380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3f15e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 0x7f8f380af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3f15e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f380b4f20 con 0x7f8f380b4b10 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3f15e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8f380b3a70 con 0x7f8f380d2c50 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3f15e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8f380b3d50 con 0x7f8f380cfc60 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f2ffff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 0x7f8f380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3d6fb640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 0x7f8f380af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f2ffff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 0x7f8f380d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:35058/0 (socket says 172.21.15.146:35058) 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f3d6fb640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 0x7f8f380af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35990/0 (socket says 172.21.15.146:35990) 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f2ffff640 1 -- 172.21.15.146:0/1637479327 learned_addr learned my addr 172.21.15.146:0/1637479327 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f2ffff640 1 -- 172.21.15.146:0/1637479327 >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 msgr2=0x7f8f380af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.933+0000 7f8f2ffff640 1 --2- 172.21.15.146:0/1637479327 >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 0x7f8f380af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f2ffff640 1 -- 172.21.15.146:0/1637479327 >> v1:172.21.15.177:6789/0 conn(0x7f8f380b4b10 legacy=0x7f8f380cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f2ffff640 1 -- 172.21.15.146:0/1637479327 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f380b4030 con 0x7f8f380cfc60 2024-04-25T15:20:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f3d6fb640 1 --2- 172.21.15.146:0/1637479327 >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 0x7f8f380af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f2ffff640 1 --2- 172.21.15.146:0/1637479327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 0x7f8f380d2710 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009870 tx=0x7f8f2802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3b01cc7afce878 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f2f7fe640 1 -- 172.21.15.146:0/1637479327 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f28037070 con 0x7f8f380cfc60 2024-04-25T15:20:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f2f7fe640 1 -- 172.21.15.146:0/1637479327 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f28032440 con 0x7f8f380cfc60 2024-04-25T15:20:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.934+0000 7f8f2f7fe640 1 -- 172.21.15.146:0/1637479327 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f28032810 con 0x7f8f380cfc60 2024-04-25T15:20:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 -- 172.21.15.146:0/1637479327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 msgr2=0x7f8f380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/1637479327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 0x7f8f380d2710 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009870 tx=0x7f8f2802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 -- 172.21.15.146:0/1637479327 shutdown_connections 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/1637479327 >> v2:172.21.15.177:3300/0 conn(0x7f8f380d2c50 0x7f8f380af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/1637479327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380cfc60 0x7f8f380d2710 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 -- 172.21.15.146:0/1637479327 >> 172.21.15.146:0/1637479327 conn(0x7f8f380b73b0 msgr2=0x7f8f380b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 -- 172.21.15.146:0/1637479327 shutdown_connections 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.935+0000 7f8f3f15e640 1 -- 172.21.15.146:0/1637479327 wait complete. 2024-04-25T15:20:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.937+0000 7f8f3f15e640 1 Processor -- start 2024-04-25T15:20:23.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.937+0000 7f8f3f15e640 1 -- start start 2024-04-25T15:20:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.938+0000 7f8f3f15e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 0x7f8f381607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.938+0000 7f8f3f15e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f380d2c50 0x7f8f3814e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.938+0000 7f8f3f15e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380c7600 0x7f8f3815bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.938+0000 7f8f3f15e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8f380d3060 con 0x7f8f380b4b10 2024-04-25T15:20:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.938+0000 7f8f3f15e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f380b45f0 con 0x7f8f380d2c50 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.938+0000 7f8f3f15e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8f380cefc0 con 0x7f8f380c7600 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 0x7f8f381607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 0x7f8f381607d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35302/0 (socket says 172.21.15.146:35302) 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 -- 172.21.15.146:0/4196892860 learned_addr learned my addr 172.21.15.146:0/4196892860 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f2ffff640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f380d2c50 0x7f8f3814e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3d6fb640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380c7600 0x7f8f3815bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 -- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380c7600 msgr2=0x7f8f3815bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380c7600 0x7f8f3815bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 -- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f380d2c50 msgr2=0x7f8f3814e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f380d2c50 0x7f8f3814e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 -- 172.21.15.146:0/4196892860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f38151150 con 0x7f8f380b4b10 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3cefa640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 0x7f8f381607d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c3f0 tx=0x7f8f3400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:23.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f340052d0 con 0x7f8f380b4b10 2024-04-25T15:20:23.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f34005490 con 0x7f8f380b4b10 2024-04-25T15:20:23.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34006a60 con 0x7f8f380b4b10 2024-04-25T15:20:23.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f380c6560 con 0x7f8f380b4b10 2024-04-25T15:20:23.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.939+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f380c6b20 con 0x7f8f380b4b10 2024-04-25T15:20:23.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.940+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f380d26f0 con 0x7f8f380b4b10 2024-04-25T15:20:23.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.941+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8f34003640 con 0x7f8f380b4b10 2024-04-25T15:20:23.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.941+0000 7f8f2d7fa640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f240744e0 0x7f8f240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:23.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.942+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8f340948b0 con 0x7f8f380b4b10 2024-04-25T15:20:23.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.942+0000 7f8f2ffff640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f240744e0 0x7f8f240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:23.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.942+0000 7f8f2ffff640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f240744e0 0x7f8f240769f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8f28034000 tx=0x7f8f2802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:23.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:23.946+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f380d26f0 con 0x7f8f380b4b10 2024-04-25T15:20:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.158+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8f3806a0e0 con 0x7f8f380b4b10 2024-04-25T15:20:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.159+0000 7f8f2d7fa640 1 -- 172.21.15.146:0/4196892860 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f8f3405e740 con 0x7f8f380b4b10 2024-04-25T15:20:24.161 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f240744e0 msgr2=0x7f8f240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f240744e0 0x7f8f240769f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8f28034000 tx=0x7f8f2802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 msgr2=0x7f8f381607d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 0x7f8f381607d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c3f0 tx=0x7f8f3400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 shutdown_connections 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f240744e0 0x7f8f240769f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8f380c7600 0x7f8f3815bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f380d2c50 0x7f8f3814e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:24.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 --2- 172.21.15.146:0/4196892860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380b4b10 0x7f8f381607d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:24.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 >> 172.21.15.146:0/4196892860 conn(0x7f8f380b73b0 msgr2=0x7f8f3806e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:24.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 shutdown_connections 2024-04-25T15:20:24.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:24.161+0000 7f8f3f15e640 1 -- 172.21.15.146:0/4196892860 wait complete. 2024-04-25T15:20:24.174 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:20:24.174 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:20:24.174 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:20:24.174 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:20:24.175 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:24.175 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:20:24.175 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:20:24.175 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:20:24.175 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:20:24.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:24.176 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:20:24.195 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:24.195 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:24.195 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:24.196 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:24.196 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:24.196 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:24.196 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:24.196 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:24.197 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:20:24.197 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:24.197 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:24.197 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:20:24.197 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:24.197 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:24.198 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:20:24.198 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:24.198 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:20:24.198 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42710sec preferred_lft 42710sec 2024-04-25T15:20:24.198 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:20:24.198 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:24.198 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add name ceph-brx type bridge 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip addr flush dev ceph-brx 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set ceph-brx up 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T15:20:24.199 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:24.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:24.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:24.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:24.393 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:20:24.417 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:20:24.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:24.419 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:20:24.488 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:20:24.488 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:20:24.488 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:20:24.488 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:20:24.488 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:20:24.489 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:24.489 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:24.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:20:24.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:20:24.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:20:24.489 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:24.576 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:24.751 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:24.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:24.758 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:20:24.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:24.773 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list-id 2024-04-25T15:20:24.828 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:24.829 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:24.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:24.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T15:20:24.829 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:24.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:24.939 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:24.946 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T15:20:24.946 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:24.995 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:25.064 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:25.070 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:25.070 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:25.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-04-25T15:20:25.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T15:20:25.071 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:25.118 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:25.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:25.145 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-25T15:20:25.145 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:25.146 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:25.160 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:20:25.160 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:25.161 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:25.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:20:25.279 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:25.335 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:20:25.335 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:20:25.336 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:20:25.337 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:20:25.338 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:20:25.338 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:20:25.338 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:25.338 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:25.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:25.338 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:20:25.390 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:20:25.391 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-25T15:20:25.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:20:25.459 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:20:25.517 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:25.518 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:25.519 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:25.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:25.520 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:20:25.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:2024-04-25T15:20:25.526+0000 7f21f4ad2480 -1 init, newargv = 0x7f21e4005700 newargc=15 2024-04-25T15:20:25.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[40874]: starting ceph client 2024-04-25T15:20:25.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[40874]: starting fuse 2024-04-25T15:20:25.543 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:20:25.543 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:25.543 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:20:25.544 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:20:25.545 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:20:25.545 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:25.545 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.1")) 2024-04-25T15:20:25.545 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:20:25.661 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.1.40874.asok 2024-04-25T15:20:25.664 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:25.672 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.40874.asok 2024-04-25T15:20:25.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:25.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.40874.asok status 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "1", 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "40874", 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/" 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:25.815 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 4628, 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 4628 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 3106257837 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:20:25.816 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 3106257837 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.4628 192.168.0.1:0/3106257837", 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.1:0/3106257837", 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 1, 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 8, 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 25, 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:20:25.817 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:20:25.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:20:25.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.945+0000 7f09b2005640 1 Processor -- start 2024-04-25T15:20:25.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.945+0000 7f09b2005640 1 -- start start 2024-04-25T15:20:25.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09b2005640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09ac0d4f60 0x7f09ac0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:25.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09b2005640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac06d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:25.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09b2005640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09ac0b9200 con 0x7f09ac0d0020 2024-04-25T15:20:25.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09b2005640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f09ac06c550 con 0x7f09ac0d4f60 2024-04-25T15:20:25.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09b2005640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac06c830 con 0x7f09ac06ccc0 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09ab7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09ac0d4f60 0x7f09ac0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09ab7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09ac0d4f60 0x7f09ac0d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35308/0 (socket says 172.21.15.146:35308) 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.946+0000 7f09ab7fe640 1 -- 172.21.15.146:0/3416313916 learned_addr learned my addr 172.21.15.146:0/3416313916 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09abfff640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac06d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09ab7fe640 1 -- 172.21.15.146:0/3416313916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 msgr2=0x7f09ac06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09ab7fe640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac06d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09ab7fe640 1 -- 172.21.15.146:0/3416313916 >> v1:172.21.15.177:6789/0 conn(0x7f09ac0d0020 legacy=0x7f09ac0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09ab7fe640 1 -- 172.21.15.146:0/3416313916 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac0d1070 con 0x7f09ac0d4f60 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09abfff640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac06d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.947+0000 7f09ab7fe640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d4f60 0x7f09ac0d5370 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f099c009870 tx=0x7f099c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97d409df648f740 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09aa7fc640 1 -- 172.21.15.146:0/3416313916 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c037070 con 0x7f09ac0d4f60 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09aa7fc640 1 -- 172.21.15.146:0/3416313916 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099c02ae10 con 0x7f09ac0d4f60 2024-04-25T15:20:25.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09aa7fc640 1 -- 172.21.15.146:0/3416313916 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c032660 con 0x7f09ac0d4f60 2024-04-25T15:20:25.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 -- 172.21.15.146:0/3416313916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d4f60 msgr2=0x7f09ac0d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:25.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d4f60 0x7f09ac0d5370 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f099c009870 tx=0x7f099c02a100 comp rx=0 tx=0).stop 2024-04-25T15:20:25.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 -- 172.21.15.146:0/3416313916 shutdown_connections 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac06d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 --2- 172.21.15.146:0/3416313916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d4f60 0x7f09ac0d5370 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 -- 172.21.15.146:0/3416313916 >> 172.21.15.146:0/3416313916 conn(0x7f09ac0d2450 msgr2=0x7f09ac0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 -- 172.21.15.146:0/3416313916 shutdown_connections 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.948+0000 7f09b2005640 1 -- 172.21.15.146:0/3416313916 wait complete. 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.950+0000 7f09b2005640 1 Processor -- start 2024-04-25T15:20:25.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.950+0000 7f09b2005640 1 -- start start 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 0x7f09ac0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09ac0d4f60 0x7f09ac15ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac0b5150 con 0x7f09ac0d0020 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac0d15f0 con 0x7f09ac0d4f60 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac0b52d0 con 0x7f09ac06ccc0 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 0x7f09ac0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 0x7f09ac0cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35312/0 (socket says 172.21.15.146:35312) 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 -- 172.21.15.146:0/643450377 learned_addr learned my addr 172.21.15.146:0/643450377 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09abfff640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09ac0d4f60 0x7f09ac15ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:25.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09ab7fe640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 -- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 msgr2=0x7f09ac0ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac0ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 -- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09ac0d4f60 msgr2=0x7f09ac15ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09ac0d4f60 0x7f09ac15ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 -- 172.21.15.146:0/643450377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac0c9630 con 0x7f09ac0d0020 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09aaffd640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 0x7f09ac0cf0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f09a000b420 tx=0x7f09a000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0011830 con 0x7f09ac0d0020 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a0016070 con 0x7f09ac0d0020 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.952+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac0c9910 con 0x7f09ac0d0020 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.953+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0011c10 con 0x7f09ac0d0020 2024-04-25T15:20:25.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.953+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac0c9ed0 con 0x7f09ac0d0020 2024-04-25T15:20:25.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.953+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09ac06a0e0 con 0x7f09ac0d0020 2024-04-25T15:20:25.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.954+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f09a001b460 con 0x7f09ac0d0020 2024-04-25T15:20:25.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.955+0000 7f0993fff640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09980744e0 0x7f09980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:25.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.955+0000 7f09ab7fe640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09980744e0 0x7f09980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:25.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.955+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f09a0094df0 con 0x7f09ac0d0020 2024-04-25T15:20:25.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.955+0000 7f09ab7fe640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09980744e0 0x7f09980769f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f099c034000 tx=0x7f099c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:25.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:25.959+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09a0020020 con 0x7f09ac0d0020 2024-04-25T15:20:26.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.175+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f09ac0d4100 con 0x7f09ac0d0020 2024-04-25T15:20:26.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.175+0000 7f0993fff640 1 -- 172.21.15.146:0/643450377 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f09a005ec80 con 0x7f09ac0d0020 2024-04-25T15:20:26.177 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09980744e0 msgr2=0x7f09980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09980744e0 0x7f09980769f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f099c034000 tx=0x7f099c02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 msgr2=0x7f09ac0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 0x7f09ac0cf0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f09a000b420 tx=0x7f09a000b8e0 comp rx=0 tx=0).stop 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 shutdown_connections 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09980744e0 0x7f09980769f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09ac0d4f60 0x7f09ac15ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09ac0d0020 0x7f09ac0cf0d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:26.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 --2- 172.21.15.146:0/643450377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09ac06ccc0 0x7f09ac0ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:26.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 >> 172.21.15.146:0/643450377 conn(0x7f09ac0d2450 msgr2=0x7f09ac0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:26.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 shutdown_connections 2024-04-25T15:20:26.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:26.177+0000 7f09b2005640 1 -- 172.21.15.146:0/643450377 wait complete. 2024-04-25T15:20:26.190 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:20:26.190 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:20:26.190 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:20:26.190 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T15:20:26.190 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:26.190 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:20:26.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:20:26.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:20:26.191 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:20:26.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.191 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:20:26.207 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42708sec preferred_lft 42708sec 2024-04-25T15:20:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link tentative 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.210 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:20:26.270 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:20:26.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.271 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:20:26.327 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:20:26.327 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:20:26.328 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:20:26.328 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:20:26.328 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:20:26.328 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:26.328 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:26.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:20:26.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:20:26.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:20:26.328 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:26.417 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:26.446 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:26.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.452 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:20:26.467 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:20:26.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.468 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list-id 2024-04-25T15:20:26.523 INFO:teuthology.orchestra.run.smithi177.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-25T15:20:26.523 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:26.523 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:26.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:20:26.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-25T15:20:26.523 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:26.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:26.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:26.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.638 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether b6:80:7b:41:cb:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:20:26.664 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:20:26.665 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.665 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::b480:7bff:fe41:cb09/64 scope link 2024-04-25T15:20:26.665 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:26.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T15:20:26.730 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:20:26.730 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:20:26.730 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:20:26.730 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.730 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether b6:80:7b:41:cb:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::b480:7bff:fe41:cb09/64 scope link 2024-04-25T15:20:26.731 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:20:26.732 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-25T15:20:26.732 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:26.732 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:26.732 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-25T15:20:26.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T15:20:26.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-25T15:20:26.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-25T15:20:26.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-25T15:20:26.733 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:26.820 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:26.873 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:26.879 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:20:26.879 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:20:26.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.1 up 2024-04-25T15:20:26.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T15:20:26.880 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:20:26.927 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:20:26.948 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:26.954 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-25T15:20:26.954 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:26.955 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:26.968 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:20:26.969 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:20:26.969 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:27.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:20:27.087 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:20:27.143 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:20:27.144 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:20:27.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.145 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:20:27.198 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:20:27.198 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-25T15:20:27.198 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-25T15:20:27.241 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:20:27.264 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:20:27.321 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:20:27.321 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:20:27.321 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:20:27.321 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:20:27.321 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:20:27.321 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:20:27.322 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:20:27.323 INFO:teuthology.orchestra.run.smithi177.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:27.324 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:20:27.324 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:20:27.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.324 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:20:27.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:2024-04-25T15:20:27.329+0000 7fa4e440d480 -1 init, newargv = 0x5586f24f1390 newargc=15 2024-04-25T15:20:27.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[41331]: starting ceph client 2024-04-25T15:20:27.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[41331]: starting fuse 2024-04-25T15:20:27.346 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:20:27.346 INFO:teuthology.orchestra.run.smithi177.stdout:44 2024-04-25T15:20:27.346 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-25T15:20:27.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:20:27.346 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:20:27.346 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:20:27.346 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:20:27.347 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:20:27.348 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:20:27.348 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:20:27.348 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:20:27.348 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:20:27.348 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.3")) 2024-04-25T15:20:27.348 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:20:27.462 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.3.41331.asok 2024-04-25T15:20:27.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:20:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:20:27.472 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.41331.asok 2024-04-25T15:20:27.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.472 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.41331.asok status 2024-04-25T15:20:27.612 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "3", 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "41331", 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/" 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:20:27.613 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 4843, 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 4843 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 2936645121 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:20:27.614 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 2936645121 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.4843 192.168.0.2:0/2936645121", 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.2:0/2936645121", 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 1, 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 8, 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 26, 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:20:27.615 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:20:27.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.621 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-04-25T15:20:27.621 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:27.643 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-04-25T15:20:27.643 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:27.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:27.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.713 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-04-25T15:20:27.713 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:27.772 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-04-25T15:20:27.772 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:27.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:27.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.840 DEBUG:teuthology.orchestra.run.smithi177:> stat --file-system '--printf=%T 2024-04-25T15:20:27.841 DEBUG:teuthology.orchestra.run.smithi177:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:27.857 INFO:teuthology.orchestra.run.smithi177.stdout:fuseblk 2024-04-25T15:20:27.857 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:27.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:27.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.922 DEBUG:teuthology.orchestra.run.smithi177:> stat --file-system '--printf=%T 2024-04-25T15:20:27.922 DEBUG:teuthology.orchestra.run.smithi177:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:27.977 INFO:teuthology.orchestra.run.smithi177.stdout:fuseblk 2024-04-25T15:20:27.978 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:27.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:27.978 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:28.042 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-25T15:20:28.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.179+0000 7fae0b227640 1 Processor -- start 2024-04-25T15:20:28.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.180+0000 7fae0b227640 1 -- start start 2024-04-25T15:20:28.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae0b227640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae040715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae0b227640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fae04160000 0x7fae0406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae0b227640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fae04070190 0x7fae040ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae0b227640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fae040705a0 con 0x7fae04160000 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae0b227640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fae04071bc0 con 0x7fae04070190 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae0b227640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fae040d1b00 con 0x7fae040b7350 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fae04160000 0x7fae0406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fae04160000 0x7fae0406fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35328/0 (socket says 172.21.15.146:35328) 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 -- 172.21.15.146:0/3943143176 learned_addr learned my addr 172.21.15.146:0/3943143176 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae097c4640 1 --2- 172.21.15.146:0/3943143176 >> v2:172.21.15.177:3300/0 conn(0x7fae04070190 0x7fae040ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fae08fc3640 1 --2- 172.21.15.146:0/3943143176 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae040715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 -- 172.21.15.146:0/3943143176 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 msgr2=0x7fae040715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 --2- 172.21.15.146:0/3943143176 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae040715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 -- 172.21.15.146:0/3943143176 >> v2:172.21.15.177:3300/0 conn(0x7fae04070190 msgr2=0x7fae040ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 --2- 172.21.15.146:0/3943143176 >> v2:172.21.15.177:3300/0 conn(0x7fae04070190 0x7fae040ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.181+0000 7fadfbfff640 1 -- 172.21.15.146:0/3943143176 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae040d1de0 con 0x7fae04160000 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.182+0000 7fadfbfff640 1 --2- 172.21.15.146:0/3943143176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae0406fc50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fae04073b40 tx=0x7fadf402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7dcd2fc65efae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.182+0000 7fadfb7fe640 1 -- 172.21.15.146:0/3943143176 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4037070 con 0x7fae04160000 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.182+0000 7fadfb7fe640 1 -- 172.21.15.146:0/3943143176 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf4032440 con 0x7fae04160000 2024-04-25T15:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.182+0000 7fadfb7fe640 1 -- 172.21.15.146:0/3943143176 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf4032810 con 0x7fae04160000 2024-04-25T15:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 -- 172.21.15.146:0/3943143176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 msgr2=0x7fae0406fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 --2- 172.21.15.146:0/3943143176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae0406fc50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fae04073b40 tx=0x7fadf402a140 comp rx=0 tx=0).stop 2024-04-25T15:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 -- 172.21.15.146:0/3943143176 shutdown_connections 2024-04-25T15:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 --2- 172.21.15.146:0/3943143176 >> v2:172.21.15.177:3300/0 conn(0x7fae04070190 0x7fae040ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 --2- 172.21.15.146:0/3943143176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae0406fc50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 --2- 172.21.15.146:0/3943143176 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae040715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 -- 172.21.15.146:0/3943143176 >> 172.21.15.146:0/3943143176 conn(0x7fae0409b2b0 msgr2=0x7fae040b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 -- 172.21.15.146:0/3943143176 shutdown_connections 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.183+0000 7fae0b227640 1 -- 172.21.15.146:0/3943143176 wait complete. 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.185+0000 7fae0b227640 1 Processor -- start 2024-04-25T15:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.185+0000 7fae0b227640 1 -- start start 2024-04-25T15:20:28.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.186+0000 7fae0b227640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae04070190 0x7fae0409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.186+0000 7fae0b227640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae0409f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.186+0000 7fae0b227640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae041f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.186+0000 7fae0b227640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fae04070980 con 0x7fae04160000 2024-04-25T15:20:28.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.186+0000 7fae0b227640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae040d23a0 con 0x7fae04070190 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.186+0000 7fae0b227640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fae04160410 con 0x7fae040b7350 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae041f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae08fc3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae04070190 0x7fae0409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae041f5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35334/0 (socket says 172.21.15.146:35334) 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 -- 172.21.15.146:0/158680069 learned_addr learned my addr 172.21.15.146:0/158680069 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fadfbfff640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae0409f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 -- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 msgr2=0x7fae0409f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae0409f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 -- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae04070190 msgr2=0x7fae0409e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae04070190 0x7fae0409e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 -- 172.21.15.146:0/158680069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae040c5ad0 con 0x7fae04160000 2024-04-25T15:20:28.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae08fc3640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae04070190 0x7fae0409e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:28.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae097c4640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae041f5f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00ef00 tx=0x7fadfc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc018020 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadfc00bcc0 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0409eb20 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc014640 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.187+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae040c6070 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.188+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae0406a0e0 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.189+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fadfc005d10 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.189+0000 7fadf97fa640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadf00744b0 0x7fadf00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.190+0000 7fae08fc3640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadf00744b0 0x7fadf00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.190+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fadfc095af0 con 0x7fae04160000 2024-04-25T15:20:28.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.190+0000 7fae08fc3640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadf00744b0 0x7fadf00769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae040745d0 tx=0x7fae00009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.194+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadfc022070 con 0x7fae04160000 2024-04-25T15:20:28.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.372+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae0406eb90 con 0x7fae04160000 2024-04-25T15:20:28.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.373+0000 7fadf97fa640 1 -- 172.21.15.146:0/158680069 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7fadfc05f840 con 0x7fae04160000 2024-04-25T15:20:28.375 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:28.375 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadf00744b0 msgr2=0x7fadf00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadf00744b0 0x7fadf00769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae040745d0 tx=0x7fae00009190 comp rx=0 tx=0).stop 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 msgr2=0x7fae041f5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae041f5f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00ef00 tx=0x7fadfc00b480 comp rx=0 tx=0).stop 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 shutdown_connections 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadf00744b0 0x7fadf00769c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae04160000 0x7fae041f5f10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae040b7350 0x7fae0409f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 --2- 172.21.15.146:0/158680069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae04070190 0x7fae0409e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 >> 172.21.15.146:0/158680069 conn(0x7fae0409b2b0 msgr2=0x7fae040b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 shutdown_connections 2024-04-25T15:20:28.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.375+0000 7fae0b227640 1 -- 172.21.15.146:0/158680069 wait complete. 2024-04-25T15:20:28.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:28.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:28.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.501+0000 7fb7e0db2640 1 Processor -- start 2024-04-25T15:20:28.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.501+0000 7fb7e0db2640 1 -- start start 2024-04-25T15:20:28.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.502+0000 7fb7e0db2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7dc0b2cd0 0x7fb7dc0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.502+0000 7fb7e0db2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc0d3300 0x7fb7dc071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.502+0000 7fb7e0db2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7dc071f70 con 0x7fb7dc071b60 2024-04-25T15:20:28.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.502+0000 7fb7e0db2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb7dc0d2260 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.502+0000 7fb7e0db2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc0d2540 con 0x7fb7dc0d3300 2024-04-25T15:20:28.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7dc0b2cd0 0x7fb7dc0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7d9d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc0d3300 0x7fb7dc071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7dc0b2cd0 0x7fb7dc0d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:36042/0 (socket says 172.21.15.146:36042) 2024-04-25T15:20:28.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 -- 172.21.15.146:0/727601202 learned_addr learned my addr 172.21.15.146:0/727601202 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:28.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 -- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc0d3300 msgr2=0x7fb7dc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 --2- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc0d3300 0x7fb7dc071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 -- 172.21.15.146:0/727601202 >> v1:172.21.15.177:6789/0 conn(0x7fb7dc071b60 legacy=0x7fb7dc08cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7da59c640 1 -- 172.21.15.146:0/727601202 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc0d2820 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.503+0000 7fb7d9d9b640 1 --2- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc0d3300 0x7fb7dc071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.504+0000 7fb7da59c640 1 --2- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc0d1c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009850 tx=0x7fb7cc02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5df6b110662015f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.504+0000 7fb7d959a640 1 -- 172.21.15.146:0/727601202 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc004030 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.504+0000 7fb7d959a640 1 -- 172.21.15.146:0/727601202 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7cc0041d0 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.504+0000 7fb7d959a640 1 -- 172.21.15.146:0/727601202 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc003610 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 -- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0b2cd0 msgr2=0x7fb7dc0d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc0d1c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009850 tx=0x7fb7cc02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 -- 172.21.15.146:0/727601202 shutdown_connections 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc0d3300 0x7fb7dc071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/727601202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc0d1c80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 -- 172.21.15.146:0/727601202 >> 172.21.15.146:0/727601202 conn(0x7fb7dc0b52e0 msgr2=0x7fb7dc0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:28.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 -- 172.21.15.146:0/727601202 shutdown_connections 2024-04-25T15:20:28.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.505+0000 7fb7e0db2640 1 -- 172.21.15.146:0/727601202 wait complete. 2024-04-25T15:20:28.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.507+0000 7fb7e0db2640 1 Processor -- start 2024-04-25T15:20:28.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.507+0000 7fb7e0db2640 1 -- start start 2024-04-25T15:20:28.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7e0db2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc071b60 0x7fb7dc0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7e0db2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7e0db2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0d3300 0x7fb7dc0ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7e0db2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc0cfb70 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7e0db2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc0d2de0 con 0x7fb7dc0d3300 2024-04-25T15:20:28.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7e0db2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc08dbf0 con 0x7fb7dc071b60 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35348/0 (socket says 172.21.15.146:35348) 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 -- 172.21.15.146:0/453547612 learned_addr learned my addr 172.21.15.146:0/453547612 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7da59c640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc071b60 0x7fb7dc0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7dad9d640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0d3300 0x7fb7dc0ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 -- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc071b60 msgr2=0x7fb7dc0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc071b60 0x7fb7dc0cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 -- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0d3300 msgr2=0x7fb7dc0ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.508+0000 7fb7d9d9b640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0d3300 0x7fb7dc0ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7d9d9b640 1 -- 172.21.15.146:0/453547612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc0c84b0 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7d9d9b640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc151ce0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb7d000e960 tx=0x7fb7d000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0017070 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d000ca00 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7dc0c8790 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d000cdd0 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.509+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7dc1527e0 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.510+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7dc0cad40 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.511+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb7d001a070 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.511+0000 7fb7caffd640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb7bc0744e0 0x7fb7bc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.511+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fb7d0095440 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.511+0000 7fb7da59c640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb7bc0744e0 0x7fb7bc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.511+0000 7fb7da59c640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb7bc0744e0 0x7fb7bc0769f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc033000 tx=0x7fb7cc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.516+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7d005f190 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.695+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb7dc06a0e0 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.695+0000 7fb7caffd640 1 -- 172.21.15.146:0/453547612 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fb7d0063490 con 0x7fb7dc0b2cd0 2024-04-25T15:20:28.697 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb7bc0744e0 msgr2=0x7fb7bc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb7bc0744e0 0x7fb7bc0769f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc033000 tx=0x7fb7cc02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 msgr2=0x7fb7dc151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc151ce0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb7d000e960 tx=0x7fb7d000ee20 comp rx=0 tx=0).stop 2024-04-25T15:20:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 shutdown_connections 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb7bc0744e0 0x7fb7bc0769f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7dc0d3300 0x7fb7dc0ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7dc0b2cd0 0x7fb7dc151ce0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 --2- 172.21.15.146:0/453547612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7dc071b60 0x7fb7dc0cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 >> 172.21.15.146:0/453547612 conn(0x7fb7dc0b52e0 msgr2=0x7fb7dc06f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 shutdown_connections 2024-04-25T15:20:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.697+0000 7fb7e0db2640 1 -- 172.21.15.146:0/453547612 wait complete. 2024-04-25T15:20:28.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:28.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.819+0000 7f3f0f6ca640 1 Processor -- start 2024-04-25T15:20:28.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.820+0000 7f3f0f6ca640 1 -- start start 2024-04-25T15:20:28.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0f6ca640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f080b2cd0 0x7f3f080d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0f6ca640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f080d2260 con 0x7f3f080d3300 2024-04-25T15:20:28.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0f6ca640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f080d2540 con 0x7f3f08071d30 2024-04-25T15:20:28.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0f6ca640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3f080d2820 con 0x7f3f080b2cd0 2024-04-25T15:20:28.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0d466640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f080b2cd0 0x7f3f080d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0cc65640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3f080d3300 0x7f3f08071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:55406/0 (socket says 172.21.15.146:55406) 2024-04-25T15:20:28.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0d466640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f080b2cd0 0x7f3f080d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35364/0 (socket says 172.21.15.146:35364) 2024-04-25T15:20:28.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0cc65640 1 -- 172.21.15.146:0/1646246843 learned_addr learned my addr 172.21.15.146:0/1646246843 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:28.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0d466640 1 -- 172.21.15.146:0/1646246843 >> v1:172.21.15.177:6789/0 conn(0x7f3f08071d30 legacy=0x7f3f080cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:28.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0d466640 1 -- 172.21.15.146:0/1646246843 >> v1:172.21.15.146:6789/0 conn(0x7f3f080d3300 legacy=0x7f3f08071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3f0d466640 1 -- 172.21.15.146:0/1646246843 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f080d2b00 con 0x7f3f080b2cd0 2024-04-25T15:20:28.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.821+0000 7f3efffff640 1 -- 172.21.15.146:0/1646246843 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680094625 0 0) 0x7f3f080d2260 con 0x7f3f080d3300 2024-04-25T15:20:28.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.822+0000 7f3f0d466640 1 --2- 172.21.15.146:0/1646246843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080b2cd0 0x7f3f080d1c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009de0 tx=0x7f3f0402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b473eb3a7ae3b4b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.822+0000 7f3efffff640 1 -- 172.21.15.146:0/1646246843 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0402ae60 con 0x7f3f080b2cd0 2024-04-25T15:20:28.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.822+0000 7f3efffff640 1 -- 172.21.15.146:0/1646246843 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f04007740 con 0x7f3f080b2cd0 2024-04-25T15:20:28.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.822+0000 7f3efffff640 1 -- 172.21.15.146:0/1646246843 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04007b10 con 0x7f3f080b2cd0 2024-04-25T15:20:28.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/1646246843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080b2cd0 msgr2=0x7f3f080d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/1646246843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080b2cd0 0x7f3f080d1c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009de0 tx=0x7f3f0402a680 comp rx=0 tx=0).stop 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/1646246843 shutdown_connections 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/1646246843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080b2cd0 0x7f3f080d1c80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/1646246843 >> 172.21.15.146:0/1646246843 conn(0x7f3f080b52e0 msgr2=0x7f3f080b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/1646246843 shutdown_connections 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.823+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/1646246843 wait complete. 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.825+0000 7f3f0f6ca640 1 Processor -- start 2024-04-25T15:20:28.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.825+0000 7f3f0f6ca640 1 -- start start 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0f6ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f080b2cd0 0x7f3f0814d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0f6ca640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 0x7f3f081e6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0f6ca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f080c8f90 0x7f3f081e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0f6ca640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0808dbf0 con 0x7f3f080d3300 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0f6ca640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3f080cfb70 con 0x7f3f080c8f90 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0f6ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0808dd70 con 0x7f3f080b2cd0 2024-04-25T15:20:28.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 0x7f3f081e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 0x7f3f081e6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35366/0 (socket says 172.21.15.146:35366) 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 -- 172.21.15.146:0/3519933521 learned_addr learned my addr 172.21.15.146:0/3519933521 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0dc67640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f080c8f90 0x7f3f081e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 -- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f080b2cd0 msgr2=0x7f3f0814d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f080b2cd0 0x7f3f0814d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 -- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f080c8f90 msgr2=0x7f3f081e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f080c8f90 0x7f3f081e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0cc65640 1 -- 172.21.15.146:0/3519933521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f080c9700 con 0x7f3f080d3300 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.826+0000 7f3f0dc67640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f080c8f90 0x7f3f081e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.827+0000 7f3f0cc65640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 0x7f3f081e6750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c3f0 tx=0x7f3ef800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.827+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80074e0 con 0x7f3f080d3300 2024-04-25T15:20:28.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.827+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef80047e0 con 0x7f3f080d3300 2024-04-25T15:20:28.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.827+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f080c99e0 con 0x7f3f080d3300 2024-04-25T15:20:28.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.827+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8004c10 con 0x7f3f080d3300 2024-04-25T15:20:28.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.827+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0814ce40 con 0x7f3f080d3300 2024-04-25T15:20:28.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.828+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f080cfcf0 con 0x7f3f080d3300 2024-04-25T15:20:28.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.828+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3ef8011070 con 0x7f3f080d3300 2024-04-25T15:20:28.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.829+0000 7f3efdffb640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3eec0744b0 0x7f3eec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:28.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.829+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f3ef8095150 con 0x7f3f080d3300 2024-04-25T15:20:28.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.829+0000 7f3f0d466640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3eec0744b0 0x7f3eec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:28.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.829+0000 7f3f0d466640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3eec0744b0 0x7f3eec0769c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f04034000 tx=0x7f3f0402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:28.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:28.833+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f080cfcf0 con 0x7f3f080d3300 2024-04-25T15:20:29.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.010+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3f080699b0 con 0x7f3f080d3300 2024-04-25T15:20:29.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.010+0000 7f3efdffb640 1 -- 172.21.15.146:0/3519933521 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f3f080699b0 con 0x7f3f080d3300 2024-04-25T15:20:29.012 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:29.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3eec0744b0 msgr2=0x7f3eec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3eec0744b0 0x7f3eec0769c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f04034000 tx=0x7f3f0402e000 comp rx=0 tx=0).stop 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 msgr2=0x7f3f081e6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 0x7f3f081e6750 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c3f0 tx=0x7f3ef800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 shutdown_connections 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3eec0744b0 0x7f3eec0769c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f080c8f90 0x7f3f081e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f080d3300 0x7f3f081e6750 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 --2- 172.21.15.146:0/3519933521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f080b2cd0 0x7f3f0814d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 >> 172.21.15.146:0/3519933521 conn(0x7f3f080b52e0 msgr2=0x7f3f080b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 shutdown_connections 2024-04-25T15:20:29.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.012+0000 7f3f0f6ca640 1 -- 172.21.15.146:0/3519933521 wait complete. 2024-04-25T15:20:29.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:29.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.134+0000 7f0eb96e7640 1 Processor -- start 2024-04-25T15:20:29.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.135+0000 7f0eb96e7640 1 -- start start 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb96e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 0x7f0eb40b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb96e7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0eb40b8f30 0x7f0eb40d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb96e7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb408d590 con 0x7f0eb40d2170 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb96e7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0eb40b9340 con 0x7f0eb40b8f30 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb96e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb40cf700 con 0x7f0eb408dd50 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb2ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 0x7f0eb40b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb27fc640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0eb40b8f30 0x7f0eb40d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb2ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 0x7f0eb40b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:35140/0 (socket says 172.21.15.146:35140) 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb2ffd640 1 -- 172.21.15.146:0/676308755 learned_addr learned my addr 172.21.15.146:0/676308755 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:29.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.136+0000 7f0eb37fe640 1 --1- 172.21.15.146:0/676308755 >> v1:172.21.15.177:6789/0 conn(0x7f0eb40d2170 0x7f0eb40b32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:46418/0 (socket says 172.21.15.146:46418) 2024-04-25T15:20:29.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb2ffd640 1 -- 172.21.15.146:0/676308755 >> v2:172.21.15.177:3300/0 conn(0x7f0eb40b8f30 msgr2=0x7f0eb40d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:29.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb2ffd640 1 --2- 172.21.15.146:0/676308755 >> v2:172.21.15.177:3300/0 conn(0x7f0eb40b8f30 0x7f0eb40d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb2ffd640 1 -- 172.21.15.146:0/676308755 >> v1:172.21.15.177:6789/0 conn(0x7f0eb40d2170 legacy=0x7f0eb40b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb1ffb640 1 -- 172.21.15.146:0/676308755 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2507637202 0 0) 0x7f0eb408d590 con 0x7f0eb40d2170 2024-04-25T15:20:29.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb2ffd640 1 -- 172.21.15.146:0/676308755 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb40d27d0 con 0x7f0eb408dd50 2024-04-25T15:20:29.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb27fc640 1 --2- 172.21.15.146:0/676308755 >> v2:172.21.15.177:3300/0 conn(0x7f0eb40b8f30 0x7f0eb40d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.137+0000 7f0eb2ffd640 1 --2- 172.21.15.146:0/676308755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 0x7f0eb40b4ca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009e50 tx=0x7f0ea402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a48c53355f3be98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb1ffb640 1 -- 172.21.15.146:0/676308755 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4036070 con 0x7f0eb408dd50 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb1ffb640 1 -- 172.21.15.146:0/676308755 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ea4031600 con 0x7f0eb408dd50 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb1ffb640 1 -- 172.21.15.146:0/676308755 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40319d0 con 0x7f0eb408dd50 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 -- 172.21.15.146:0/676308755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 msgr2=0x7f0eb40b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/676308755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 0x7f0eb40b4ca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009e50 tx=0x7f0ea402a710 comp rx=0 tx=0).stop 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 -- 172.21.15.146:0/676308755 shutdown_connections 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/676308755 >> v2:172.21.15.177:3300/0 conn(0x7f0eb40b8f30 0x7f0eb40d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/676308755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb408dd50 0x7f0eb40b4ca0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 -- 172.21.15.146:0/676308755 >> 172.21.15.146:0/676308755 conn(0x7f0eb40b5430 msgr2=0x7f0eb40b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 -- 172.21.15.146:0/676308755 shutdown_connections 2024-04-25T15:20:29.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.138+0000 7f0eb96e7640 1 -- 172.21.15.146:0/676308755 wait complete. 2024-04-25T15:20:29.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.140+0000 7f0eb96e7640 1 Processor -- start 2024-04-25T15:20:29.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.140+0000 7f0eb96e7640 1 -- start start 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb40b8f30 0x7f0eb406e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb40d2170 0x7f0eb41da400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 0x7f0eb41e8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb40d2580 con 0x7f0eb41e89f0 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb408d8c0 con 0x7f0eb40b8f30 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb40d2ab0 con 0x7f0eb40d2170 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 0x7f0eb41e8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 0x7f0eb41e8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35382/0 (socket says 172.21.15.146:35382) 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 -- 172.21.15.146:0/3234639408 learned_addr learned my addr 172.21.15.146:0/3234639408 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb2ffd640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb40b8f30 0x7f0eb406e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb27fc640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb40d2170 0x7f0eb41da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 -- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb40d2170 msgr2=0x7f0eb41da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb40d2170 0x7f0eb41da400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 -- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb40b8f30 msgr2=0x7f0eb406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb40b8f30 0x7f0eb406e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 -- 172.21.15.146:0/3234639408 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb41e9340 con 0x7f0eb41e89f0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb37fe640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 0x7f0eb41e8e00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c470 tx=0x7f0eac00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0074e0 con 0x7f0eb41e89f0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eac0047e0 con 0x7f0eb41e89f0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb406ec70 con 0x7f0eb41e89f0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac004bd0 con 0x7f0eb41e89f0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.142+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb406f230 con 0x7f0eb41e89f0 2024-04-25T15:20:29.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.143+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb40693b0 con 0x7f0eb41e89f0 2024-04-25T15:20:29.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.144+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0eac011070 con 0x7f0eb41e89f0 2024-04-25T15:20:29.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.144+0000 7f0e9b7fe640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ea00744e0 0x7f0ea00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.145+0000 7f0eb2ffd640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ea00744e0 0x7f0ea00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.145+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f0eac095030 con 0x7f0eb41e89f0 2024-04-25T15:20:29.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.145+0000 7f0eb2ffd640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ea00744e0 0x7f0ea00769f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4033000 tx=0x7f0ea4009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.149+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0eb40693b0 con 0x7f0eb41e89f0 2024-04-25T15:20:29.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.324+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0eb406a0e0 con 0x7f0eb41e89f0 2024-04-25T15:20:29.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.325+0000 7f0e9b7fe640 1 -- 172.21.15.146:0/3234639408 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f0eb41e9340 con 0x7f0eb41e89f0 2024-04-25T15:20:29.327 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ea00744e0 msgr2=0x7f0ea00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ea00744e0 0x7f0ea00769f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4033000 tx=0x7f0ea4009000 comp rx=0 tx=0).stop 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 msgr2=0x7f0eb41e8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 0x7f0eb41e8e00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c470 tx=0x7f0eac00c930 comp rx=0 tx=0).stop 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 shutdown_connections 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ea00744e0 0x7f0ea00769f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb41e89f0 0x7f0eb41e8e00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb40d2170 0x7f0eb41da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 --2- 172.21.15.146:0/3234639408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb40b8f30 0x7f0eb406e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 >> 172.21.15.146:0/3234639408 conn(0x7f0eb40b5430 msgr2=0x7f0eb4071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 shutdown_connections 2024-04-25T15:20:29.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.327+0000 7f0eb96e7640 1 -- 172.21.15.146:0/3234639408 wait complete. 2024-04-25T15:20:29.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:29.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.449+0000 7f3189805640 1 Processor -- start 2024-04-25T15:20:29.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.449+0000 7f3189805640 1 -- start start 2024-04-25T15:20:29.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.450+0000 7f3189805640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3184163c00 0x7f318406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.450+0000 7f3189805640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31840d0fb0 0x7f31840b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.450+0000 7f3189805640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f318406dea0 con 0x7f31840bb450 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.450+0000 7f3189805640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f31840b5230 con 0x7f31840d0fb0 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.450+0000 7f3189805640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f318406c8c0 con 0x7f3184163c00 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31837fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31840d0fb0 0x7f31840b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31827fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f31840bb450 0x7f31840d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:55422/0 (socket says 172.21.15.146:55422) 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31827fc640 1 -- 172.21.15.146:0/969231610 learned_addr learned my addr 172.21.15.146:0/969231610 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f3182ffd640 1 --2- 172.21.15.146:0/969231610 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3184163c00 0x7f318406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31837fe640 1 -- 172.21.15.146:0/969231610 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3184163c00 msgr2=0x7f318406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31837fe640 1 --2- 172.21.15.146:0/969231610 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3184163c00 0x7f318406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31837fe640 1 -- 172.21.15.146:0/969231610 >> v1:172.21.15.146:6789/0 conn(0x7f31840bb450 legacy=0x7f31840d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f31837fe640 1 -- 172.21.15.146:0/969231610 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318406cba0 con 0x7f31840d0fb0 2024-04-25T15:20:29.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.451+0000 7f3181ffb640 1 -- 172.21.15.146:0/969231610 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3781112947 0 0) 0x7f318406dea0 con 0x7f31840bb450 2024-04-25T15:20:29.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f31837fe640 1 --2- 172.21.15.146:0/969231610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31840d0fb0 0x7f31840b8fb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f317c009d90 tx=0x7f317c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b809ea4a45d5d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3181ffb640 1 -- 172.21.15.146:0/969231610 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c031600 con 0x7f31840d0fb0 2024-04-25T15:20:29.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3181ffb640 1 -- 172.21.15.146:0/969231610 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f317c0317a0 con 0x7f31840d0fb0 2024-04-25T15:20:29.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3181ffb640 1 -- 172.21.15.146:0/969231610 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c031b70 con 0x7f31840d0fb0 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3189805640 1 -- 172.21.15.146:0/969231610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31840d0fb0 msgr2=0x7f31840b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3189805640 1 --2- 172.21.15.146:0/969231610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31840d0fb0 0x7f31840b8fb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f317c009d90 tx=0x7f317c02a5d0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3189805640 1 -- 172.21.15.146:0/969231610 shutdown_connections 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3189805640 1 --2- 172.21.15.146:0/969231610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31840d0fb0 0x7f31840b8fb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3189805640 1 --2- 172.21.15.146:0/969231610 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3184163c00 0x7f318406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.452+0000 7f3189805640 1 -- 172.21.15.146:0/969231610 >> 172.21.15.146:0/969231610 conn(0x7f31840b7970 msgr2=0x7f31840b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.453+0000 7f3189805640 1 -- 172.21.15.146:0/969231610 shutdown_connections 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.453+0000 7f3189805640 1 -- 172.21.15.146:0/969231610 wait complete. 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.454+0000 7f3189805640 1 Processor -- start 2024-04-25T15:20:29.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.454+0000 7f3189805640 1 -- start start 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31840bb450 0x7f31840ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31840d0fb0 0x7f31840ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 0x7f31840cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f318406c550 con 0x7f3184163c00 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f318406d160 con 0x7f31840d0fb0 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f31840d13c0 con 0x7f31840bb450 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 0x7f31840cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 0x7f31840cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35406/0 (socket says 172.21.15.146:35406) 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 -- 172.21.15.146:0/162447666 learned_addr learned my addr 172.21.15.146:0/162447666 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3182ffd640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31840bb450 0x7f31840ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31827fc640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31840d0fb0 0x7f31840ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 -- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31840bb450 msgr2=0x7f31840ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31840bb450 0x7f31840ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 -- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31840d0fb0 msgr2=0x7f31840ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31840d0fb0 0x7f31840ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 -- 172.21.15.146:0/162447666 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31840cd2f0 con 0x7f3184163c00 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f31837fe640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 0x7f31840cee10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f317c009eb0 tx=0x7f317c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c03f070 con 0x7f3184163c00 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f317c030be0 con 0x7f3184163c00 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31840cd5d0 con 0x7f3184163c00 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.456+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c009450 con 0x7f3184163c00 2024-04-25T15:20:29.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.457+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31840cdb90 con 0x7f3184163c00 2024-04-25T15:20:29.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.458+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f318406a0e0 con 0x7f3184163c00 2024-04-25T15:20:29.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.458+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f317c003990 con 0x7f3184163c00 2024-04-25T15:20:29.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.459+0000 7f316bfff640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31700744b0 0x7f31700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.459+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f317c0b6670 con 0x7f3184163c00 2024-04-25T15:20:29.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.459+0000 7f3182ffd640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31700744b0 0x7f31700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.459+0000 7f3182ffd640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31700744b0 0x7f31700769c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3174006fd0 tx=0x7f3174008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.463+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f317c080aa0 con 0x7f3184163c00 2024-04-25T15:20:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.643+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f31840cbf90 con 0x7f3184163c00 2024-04-25T15:20:29.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.643+0000 7f316bfff640 1 -- 172.21.15.146:0/162447666 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f317c0803a0 con 0x7f3184163c00 2024-04-25T15:20:29.645 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31700744b0 msgr2=0x7f31700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31700744b0 0x7f31700769c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3174006fd0 tx=0x7f3174008040 comp rx=0 tx=0).stop 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 msgr2=0x7f31840cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 0x7f31840cee10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f317c009eb0 tx=0x7f317c004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 shutdown_connections 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31700744b0 0x7f31700769c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3184163c00 0x7f31840cee10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31840d0fb0 0x7f31840ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 --2- 172.21.15.146:0/162447666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31840bb450 0x7f31840ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 >> 172.21.15.146:0/162447666 conn(0x7f31840b7970 msgr2=0x7f31840b9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 shutdown_connections 2024-04-25T15:20:29.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.646+0000 7f3189805640 1 -- 172.21.15.146:0/162447666 wait complete. 2024-04-25T15:20:29.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:29.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.766+0000 7fedf0492640 1 Processor -- start 2024-04-25T15:20:29.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.766+0000 7fedf0492640 1 -- start start 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedf0492640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fede806ee60 0x7fede80b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedf0492640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fede81649d0 0x7fede80d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedf0492640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede80b8eb0 con 0x7fede8163eb0 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedf0492640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fede8161840 con 0x7fede806ee60 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedf0492640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fede8164de0 con 0x7fede81649d0 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fede806ee60 0x7fede80b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fede806ee60 0x7fede80b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35416/0 (socket says 172.21.15.146:35416) 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 -- 172.21.15.146:0/127414577 learned_addr learned my addr 172.21.15.146:0/127414577 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedeea2f640 1 --2- 172.21.15.146:0/127414577 >> v2:172.21.15.177:3300/0 conn(0x7fede81649d0 0x7fede80d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 -- 172.21.15.146:0/127414577 >> v2:172.21.15.177:3300/0 conn(0x7fede81649d0 msgr2=0x7fede80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 --2- 172.21.15.146:0/127414577 >> v2:172.21.15.177:3300/0 conn(0x7fede81649d0 0x7fede80d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 -- 172.21.15.146:0/127414577 >> v1:172.21.15.177:6789/0 conn(0x7fede8163eb0 legacy=0x7fede81642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.768+0000 7fedee22e640 1 -- 172.21.15.146:0/127414577 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede808ce00 con 0x7fede806ee60 2024-04-25T15:20:29.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.769+0000 7fedee22e640 1 --2- 172.21.15.146:0/127414577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede806ee60 0x7fede80b2ba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fede80b5290 tx=0x7fede402a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d61f793800467b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.769+0000 7feded22c640 1 -- 172.21.15.146:0/127414577 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede4004030 con 0x7fede806ee60 2024-04-25T15:20:29.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.769+0000 7feded22c640 1 -- 172.21.15.146:0/127414577 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fede40041d0 con 0x7fede806ee60 2024-04-25T15:20:29.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.769+0000 7feded22c640 1 -- 172.21.15.146:0/127414577 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede4003610 con 0x7fede806ee60 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 -- 172.21.15.146:0/127414577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede806ee60 msgr2=0x7fede80b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 --2- 172.21.15.146:0/127414577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede806ee60 0x7fede80b2ba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fede80b5290 tx=0x7fede402a290 comp rx=0 tx=0).stop 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 -- 172.21.15.146:0/127414577 shutdown_connections 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 --2- 172.21.15.146:0/127414577 >> v2:172.21.15.177:3300/0 conn(0x7fede81649d0 0x7fede80d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 --2- 172.21.15.146:0/127414577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede806ee60 0x7fede80b2ba0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 -- 172.21.15.146:0/127414577 >> 172.21.15.146:0/127414577 conn(0x7fede80b52e0 msgr2=0x7fede80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 -- 172.21.15.146:0/127414577 shutdown_connections 2024-04-25T15:20:29.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.770+0000 7fedf0492640 1 -- 172.21.15.146:0/127414577 wait complete. 2024-04-25T15:20:29.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.772+0000 7fedf0492640 1 Processor -- start 2024-04-25T15:20:29.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.772+0000 7fedf0492640 1 -- start start 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedf0492640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede806ee60 0x7fede80b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedf0492640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede8163eb0 0x7fede80b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedf0492640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 0x7fede815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedf0492640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fede806f440 con 0x7fede81649d0 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedf0492640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fede808d730 con 0x7fede806ee60 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedf0492640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fede808d3c0 con 0x7fede8163eb0 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 0x7fede815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 0x7fede815c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35424/0 (socket says 172.21.15.146:35424) 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 -- 172.21.15.146:0/1992766819 learned_addr learned my addr 172.21.15.146:0/1992766819 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:29.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedee22e640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede806ee60 0x7fede80b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fededa2d640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede8163eb0 0x7fede80b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 -- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede8163eb0 msgr2=0x7fede80b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede8163eb0 0x7fede80b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 -- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede806ee60 msgr2=0x7fede80b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede806ee60 0x7fede80b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedeea2f640 1 -- 172.21.15.146:0/1992766819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede80b12f0 con 0x7fede81649d0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fededa2d640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede8163eb0 0x7fede80b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.773+0000 7fedee22e640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede806ee60 0x7fede80b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.774+0000 7fedeea2f640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 0x7fede815c5c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fede000efc0 tx=0x7fede000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.774+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede0018020 con 0x7fede81649d0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.774+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fede0014480 con 0x7fede81649d0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.774+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede0009e60 con 0x7fede81649d0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.774+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede8162050 con 0x7fede81649d0 2024-04-25T15:20:29.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.774+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede8162610 con 0x7fede81649d0 2024-04-25T15:20:29.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.775+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede8161dc0 con 0x7fede81649d0 2024-04-25T15:20:29.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.775+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fede0005d10 con 0x7fede81649d0 2024-04-25T15:20:29.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.776+0000 7feddeffd640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fedcc0744b0 0x7fedcc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:29.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.776+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fede00963b0 con 0x7fede81649d0 2024-04-25T15:20:29.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.776+0000 7fedee22e640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fedcc0744b0 0x7fedcc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:29.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.776+0000 7fedee22e640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fedcc0744b0 0x7fedcc0769c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fede4033000 tx=0x7fede402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:29.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.780+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fede0022030 con 0x7fede81649d0 2024-04-25T15:20:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.956+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fede80b61f0 con 0x7fede81649d0 2024-04-25T15:20:29.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.958+0000 7feddeffd640 1 -- 172.21.15.146:0/1992766819 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7fede0010040 con 0x7fede81649d0 2024-04-25T15:20:29.960 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:29.960 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fedcc0744b0 msgr2=0x7fedcc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fedcc0744b0 0x7fedcc0769c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fede4033000 tx=0x7fede402e000 comp rx=0 tx=0).stop 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 msgr2=0x7fede815c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 0x7fede815c5c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fede000efc0 tx=0x7fede000c490 comp rx=0 tx=0).stop 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 shutdown_connections 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fedcc0744b0 0x7fedcc0769c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede81649d0 0x7fede815c5c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede8163eb0 0x7fede80b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 --2- 172.21.15.146:0/1992766819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede806ee60 0x7fede80b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 >> 172.21.15.146:0/1992766819 conn(0x7fede80b52e0 msgr2=0x7fede815feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 shutdown_connections 2024-04-25T15:20:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:29.960+0000 7fedf0492640 1 -- 172.21.15.146:0/1992766819 wait complete. 2024-04-25T15:20:29.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:29.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:30.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.082+0000 7fdd213e3640 1 Processor -- start 2024-04-25T15:20:30.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.082+0000 7fdd213e3640 1 -- start start 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd213e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c070500 0x7fdd1c0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd213e3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdd1c0d10b0 0x7fdd1c15cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd213e3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1c0b0520 con 0x7fdd1c06d560 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd213e3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c0b0850 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd213e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c070d90 con 0x7fdd1c070500 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1a7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdd1c0d10b0 0x7fdd1c15cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdd1c06d560 0x7fdd1c06d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:55434/0 (socket says 172.21.15.146:55434) 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1b7fe640 1 -- 172.21.15.146:0/3075517519 learned_addr learned my addr 172.21.15.146:0/3075517519 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1affd640 1 --2- 172.21.15.146:0/3075517519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c070500 0x7fdd1c0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1a7fc640 1 -- 172.21.15.146:0/3075517519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c070500 msgr2=0x7fdd1c0b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1a7fc640 1 --2- 172.21.15.146:0/3075517519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c070500 0x7fdd1c0b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1a7fc640 1 -- 172.21.15.146:0/3075517519 >> v1:172.21.15.146:6789/0 conn(0x7fdd1c06d560 legacy=0x7fdd1c06d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd1a7fc640 1 -- 172.21.15.146:0/3075517519 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c071070 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.084+0000 7fdd19ffb640 1 -- 172.21.15.146:0/3075517519 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342575965 0 0) 0x7fdd1c0b0520 con 0x7fdd1c06d560 2024-04-25T15:20:30.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.085+0000 7fdd1a7fc640 1 --2- 172.21.15.146:0/3075517519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c15cbb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdd100097a0 tx=0x7fdd1002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68521f26adaf67e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.085+0000 7fdd19ffb640 1 -- 172.21.15.146:0/3075517519 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10037070 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.085+0000 7fdd19ffb640 1 -- 172.21.15.146:0/3075517519 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd10032440 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.085+0000 7fdd19ffb640 1 -- 172.21.15.146:0/3075517519 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10032810 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 -- 172.21.15.146:0/3075517519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 msgr2=0x7fdd1c15cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 --2- 172.21.15.146:0/3075517519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c15cbb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdd100097a0 tx=0x7fdd1002a180 comp rx=0 tx=0).stop 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 -- 172.21.15.146:0/3075517519 shutdown_connections 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 --2- 172.21.15.146:0/3075517519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c15cbb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 --2- 172.21.15.146:0/3075517519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c070500 0x7fdd1c0b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 -- 172.21.15.146:0/3075517519 >> 172.21.15.146:0/3075517519 conn(0x7fdd1c0d1c10 msgr2=0x7fdd1c0733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 -- 172.21.15.146:0/3075517519 shutdown_connections 2024-04-25T15:20:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.086+0000 7fdd213e3640 1 -- 172.21.15.146:0/3075517519 wait complete. 2024-04-25T15:20:30.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.088+0000 7fdd213e3640 1 Processor -- start 2024-04-25T15:20:30.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.088+0000 7fdd213e3640 1 -- start start 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd213e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c06d560 0x7fdd1c1f8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd213e3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1c070500 0x7fdd1c1f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd213e3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c14b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd213e3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0b4790 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd213e3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0b4610 con 0x7fdd1c070500 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd213e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c0b4910 con 0x7fdd1c06d560 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c14b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c14b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35438/0 (socket says 172.21.15.146:35438) 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 -- 172.21.15.146:0/1655235459 learned_addr learned my addr 172.21.15.146:0/1655235459 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1affd640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c06d560 0x7fdd1c1f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1a7fc640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1c070500 0x7fdd1c1f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 -- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c06d560 msgr2=0x7fdd1c1f8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c06d560 0x7fdd1c1f8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 -- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1c070500 msgr2=0x7fdd1c1f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.089+0000 7fdd1b7fe640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1c070500 0x7fdd1c1f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd1b7fe640 1 -- 172.21.15.146:0/1655235459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c1f89a0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd1affd640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c06d560 0x7fdd1c1f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd1a7fc640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1c070500 0x7fdd1c1f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd1b7fe640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c14b7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400c3e0 tx=0x7fdd1400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14017070 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd14005ce0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1c15c380 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd140072f0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.090+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1c1512a0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.091+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1c06a0e0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.092+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdd1401a070 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.092+0000 7fdd037fe640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdd080744e0 0x7fdd080769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.092+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fdd140948b0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.092+0000 7fdd1affd640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdd080744e0 0x7fdd080769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.093+0000 7fdd1affd640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdd080744e0 0x7fdd080769f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0af9f0 tx=0x7fdd0c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.097+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd1c1f89a0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.275+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd1c0cef50 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.275+0000 7fdd037fe640 1 -- 172.21.15.146:0/1655235459 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fdd1405ece0 con 0x7fdd1c0d10b0 2024-04-25T15:20:30.277 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdd080744e0 msgr2=0x7fdd080769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdd080744e0 0x7fdd080769f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0af9f0 tx=0x7fdd0c00b040 comp rx=0 tx=0).stop 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 msgr2=0x7fdd1c14b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c14b7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400c3e0 tx=0x7fdd1400c8a0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 shutdown_connections 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdd080744e0 0x7fdd080769f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd1c0d10b0 0x7fdd1c14b7b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd1c070500 0x7fdd1c1f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.277+0000 7fdd213e3640 1 --2- 172.21.15.146:0/1655235459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd1c06d560 0x7fdd1c1f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.278+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 >> 172.21.15.146:0/1655235459 conn(0x7fdd1c0d1c10 msgr2=0x7fdd1c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:30.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.278+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 shutdown_connections 2024-04-25T15:20:30.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.278+0000 7fdd213e3640 1 -- 172.21.15.146:0/1655235459 wait complete. 2024-04-25T15:20:30.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:30.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.399+0000 7fd9be34d640 1 Processor -- start 2024-04-25T15:20:30.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.399+0000 7fd9be34d640 1 -- start start 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9be34d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9b80b2cd0 0x7fd9b80d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9be34d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd9b80d3300 0x7fd9b8071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9be34d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b8071f70 con 0x7fd9b8071b60 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9be34d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b80d2260 con 0x7fd9b80d3300 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9be34d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b80d2540 con 0x7fd9b80b2cd0 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd9b80d3300 0x7fd9b8071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd9b80d3300 0x7fd9b8071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35444/0 (socket says 172.21.15.146:35444) 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 -- 172.21.15.146:0/2633369583 learned_addr learned my addr 172.21.15.146:0/2633369583 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b7fff640 1 --2- 172.21.15.146:0/2633369583 >> v2:172.21.15.177:3300/0 conn(0x7fd9b80b2cd0 0x7fd9b80d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 -- 172.21.15.146:0/2633369583 >> v2:172.21.15.177:3300/0 conn(0x7fd9b80b2cd0 msgr2=0x7fd9b80d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 --2- 172.21.15.146:0/2633369583 >> v2:172.21.15.177:3300/0 conn(0x7fd9b80b2cd0 0x7fd9b80d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 -- 172.21.15.146:0/2633369583 >> v1:172.21.15.177:6789/0 conn(0x7fd9b8071b60 legacy=0x7fd9b808cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:30.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.401+0000 7fd9b77fe640 1 -- 172.21.15.146:0/2633369583 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b80d2820 con 0x7fd9b80d3300 2024-04-25T15:20:30.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.402+0000 7fd9b77fe640 1 --2- 172.21.15.146:0/2633369583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b8071620 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd9b808c9d0 tx=0x7fd9ac02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d76fd1767f45b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.402+0000 7fd9b6ffd640 1 -- 172.21.15.146:0/2633369583 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac037070 con 0x7fd9b80d3300 2024-04-25T15:20:30.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.402+0000 7fd9b6ffd640 1 -- 172.21.15.146:0/2633369583 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ac032440 con 0x7fd9b80d3300 2024-04-25T15:20:30.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.402+0000 7fd9b6ffd640 1 -- 172.21.15.146:0/2633369583 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac032810 con 0x7fd9b80d3300 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 -- 172.21.15.146:0/2633369583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 msgr2=0x7fd9b8071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 --2- 172.21.15.146:0/2633369583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b8071620 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd9b808c9d0 tx=0x7fd9ac02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 -- 172.21.15.146:0/2633369583 shutdown_connections 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 --2- 172.21.15.146:0/2633369583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b8071620 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 --2- 172.21.15.146:0/2633369583 >> v2:172.21.15.177:3300/0 conn(0x7fd9b80b2cd0 0x7fd9b80d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 -- 172.21.15.146:0/2633369583 >> 172.21.15.146:0/2633369583 conn(0x7fd9b80b52e0 msgr2=0x7fd9b80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 -- 172.21.15.146:0/2633369583 shutdown_connections 2024-04-25T15:20:30.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.403+0000 7fd9be34d640 1 -- 172.21.15.146:0/2633369583 wait complete. 2024-04-25T15:20:30.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.405+0000 7fd9be34d640 1 Processor -- start 2024-04-25T15:20:30.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.405+0000 7fd9be34d640 1 -- start start 2024-04-25T15:20:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9be34d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9b8071b60 0x7fd9b815e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9be34d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9b80b2cd0 0x7fd9b80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9be34d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b80c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9be34d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80cfb70 con 0x7fd9b80d3300 2024-04-25T15:20:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9be34d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80d2de0 con 0x7fd9b80b2cd0 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9be34d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b808dbf0 con 0x7fd9b8071b60 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b80c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b80c9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35458/0 (socket says 172.21.15.146:35458) 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 -- 172.21.15.146:0/3624272427 learned_addr learned my addr 172.21.15.146:0/3624272427 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9b77fe640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9b80b2cd0 0x7fd9b80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9b7fff640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9b8071b60 0x7fd9b815e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 -- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9b8071b60 msgr2=0x7fd9b815e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9b8071b60 0x7fd9b815e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 -- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9b80b2cd0 msgr2=0x7fd9b80c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9b80b2cd0 0x7fd9b80c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.406+0000 7fd9bc8ea640 1 -- 172.21.15.146:0/3624272427 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b80c77f0 con 0x7fd9b80d3300 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.407+0000 7fd9bc8ea640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b80c9b20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000b990 tx=0x7fd9b000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.407+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0019070 con 0x7fd9b80d3300 2024-04-25T15:20:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.407+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b000cba0 con 0x7fd9b80d3300 2024-04-25T15:20:30.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.407+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b8160730 con 0x7fd9b80d3300 2024-04-25T15:20:30.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.407+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0014440 con 0x7fd9b80d3300 2024-04-25T15:20:30.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.407+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b8160cf0 con 0x7fd9b80d3300 2024-04-25T15:20:30.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.408+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b806a0e0 con 0x7fd9b80d3300 2024-04-25T15:20:30.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.409+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd9b001c070 con 0x7fd9b80d3300 2024-04-25T15:20:30.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.409+0000 7fd9b4ff9640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd9a40744e0 0x7fd9a40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.409+0000 7fd9b7fff640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd9a40744e0 0x7fd9a40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.409+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fd9b0095380 con 0x7fd9b80d3300 2024-04-25T15:20:30.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.410+0000 7fd9b7fff640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd9a40744e0 0x7fd9a40769f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9b80b9490 tx=0x7fd9a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.414+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9b005f7b0 con 0x7fd9b80d3300 2024-04-25T15:20:30.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.592+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd9b8163bf0 con 0x7fd9b80d3300 2024-04-25T15:20:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.593+0000 7fd9b4ff9640 1 -- 172.21.15.146:0/3624272427 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fd9b005f0b0 con 0x7fd9b80d3300 2024-04-25T15:20:30.595 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd9a40744e0 msgr2=0x7fd9a40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd9a40744e0 0x7fd9a40769f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9b80b9490 tx=0x7fd9a8008040 comp rx=0 tx=0).stop 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 msgr2=0x7fd9b80c9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b80c9b20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000b990 tx=0x7fd9b000be50 comp rx=0 tx=0).stop 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 shutdown_connections 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd9a40744e0 0x7fd9a40769f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9b80d3300 0x7fd9b80c9b20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9b80b2cd0 0x7fd9b80c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 --2- 172.21.15.146:0/3624272427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9b8071b60 0x7fd9b815e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 >> 172.21.15.146:0/3624272427 conn(0x7fd9b80b52e0 msgr2=0x7fd9b80b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:30.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 shutdown_connections 2024-04-25T15:20:30.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.595+0000 7fd9be34d640 1 -- 172.21.15.146:0/3624272427 wait complete. 2024-04-25T15:20:30.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:30.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.718+0000 7f5638ebd640 1 Processor -- start 2024-04-25T15:20:30.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.718+0000 7f5638ebd640 1 -- start start 2024-04-25T15:20:30.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5638ebd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5634070500 0x7f56340b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5638ebd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56340d10b0 0x7f563415cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5638ebd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56340b0520 con 0x7f563406d560 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5638ebd640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f56340b0850 con 0x7f56340d10b0 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5638ebd640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5634070d90 con 0x7f5634070500 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5631d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56340d10b0 0x7f563415cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5632d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f563406d560 0x7f563406d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42828/0 (socket says 172.21.15.146:42828) 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5632d9d640 1 -- 172.21.15.146:0/3864234 learned_addr learned my addr 172.21.15.146:0/3864234 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f563259c640 1 --2- 172.21.15.146:0/3864234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5634070500 0x7f56340b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5631d9b640 1 -- 172.21.15.146:0/3864234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5634070500 msgr2=0x7f56340b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5631d9b640 1 --2- 172.21.15.146:0/3864234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5634070500 0x7f56340b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5631d9b640 1 -- 172.21.15.146:0/3864234 >> v1:172.21.15.146:6789/0 conn(0x7f563406d560 legacy=0x7f563406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f5631d9b640 1 -- 172.21.15.146:0/3864234 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5634071070 con 0x7f56340d10b0 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.720+0000 7f563159a640 1 -- 172.21.15.146:0/3864234 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250109542 0 0) 0x7f56340b0520 con 0x7f563406d560 2024-04-25T15:20:30.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.721+0000 7f5631d9b640 1 --2- 172.21.15.146:0/3864234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340d10b0 0x7f563415cbb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f562800ba40 tx=0x7f562802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6708e2e654aefa6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.721+0000 7f563159a640 1 -- 172.21.15.146:0/3864234 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628032750 con 0x7f56340d10b0 2024-04-25T15:20:30.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.721+0000 7f563159a640 1 -- 172.21.15.146:0/3864234 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5628030050 con 0x7f56340d10b0 2024-04-25T15:20:30.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.721+0000 7f563159a640 1 -- 172.21.15.146:0/3864234 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628032b10 con 0x7f56340d10b0 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 -- 172.21.15.146:0/3864234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340d10b0 msgr2=0x7f563415cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 --2- 172.21.15.146:0/3864234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340d10b0 0x7f563415cbb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f562800ba40 tx=0x7f562802c850 comp rx=0 tx=0).stop 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 -- 172.21.15.146:0/3864234 shutdown_connections 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 --2- 172.21.15.146:0/3864234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56340d10b0 0x7f563415cbb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 --2- 172.21.15.146:0/3864234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5634070500 0x7f56340b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 -- 172.21.15.146:0/3864234 >> 172.21.15.146:0/3864234 conn(0x7f56340d1c10 msgr2=0x7f56340733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 -- 172.21.15.146:0/3864234 shutdown_connections 2024-04-25T15:20:30.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.722+0000 7f5638ebd640 1 -- 172.21.15.146:0/3864234 wait complete. 2024-04-25T15:20:30.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.724+0000 7f5638ebd640 1 Processor -- start 2024-04-25T15:20:30.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.724+0000 7f5638ebd640 1 -- start start 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5638ebd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 0x7f5634158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5638ebd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634070500 0x7f563409b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5638ebd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56340d10b0 0x7f563415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5638ebd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f56340b4790 con 0x7f563406d560 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5638ebd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f56340b4610 con 0x7f5634070500 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5638ebd640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f56340b4910 con 0x7f56340d10b0 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 0x7f5634158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 0x7f5634158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51194/0 (socket says 172.21.15.146:51194) 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 -- 172.21.15.146:0/521784997 learned_addr learned my addr 172.21.15.146:0/521784997 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5631d9b640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634070500 0x7f563409b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5632d9d640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56340d10b0 0x7f563415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 -- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56340d10b0 msgr2=0x7f563415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56340d10b0 0x7f563415b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 -- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634070500 msgr2=0x7f563409b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634070500 0x7f563409b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f563259c640 1 -- 172.21.15.146:0/521784997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56341f2490 con 0x7f563406d560 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5632d9d640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56340d10b0 0x7f563415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.725+0000 7f5631d9b640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634070500 0x7f563409b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.726+0000 7f563259c640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 0x7f5634158360 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f562400d5d0 tx=0x7f562400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.726+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624004240 con 0x7f563406d560 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.726+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56240043e0 con 0x7f563406d560 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.726+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56341f2770 con 0x7f563406d560 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.726+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624003830 con 0x7f563406d560 2024-04-25T15:20:30.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.726+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56341f2d30 con 0x7f563406d560 2024-04-25T15:20:30.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.727+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563406a0e0 con 0x7f563406d560 2024-04-25T15:20:30.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.727+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f56240039d0 con 0x7f563406d560 2024-04-25T15:20:30.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.728+0000 7f5622ffd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56140744b0 0x7f56140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:30.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.728+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5624095890 con 0x7f563406d560 2024-04-25T15:20:30.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.728+0000 7f5631d9b640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56140744b0 0x7f56140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:30.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.729+0000 7f5631d9b640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56140744b0 0x7f56140769c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5628037000 tx=0x7f5628032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:30.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.733+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f562405fcc0 con 0x7f563406d560 2024-04-25T15:20:30.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.943+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f563409e030 con 0x7f563406d560 2024-04-25T15:20:30.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.943+0000 7f5622ffd640 1 -- 172.21.15.146:0/521784997 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f563409e030 con 0x7f563406d560 2024-04-25T15:20:30.945 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:30.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56140744b0 msgr2=0x7f56140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56140744b0 0x7f56140769c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5628037000 tx=0x7f5628032000 comp rx=0 tx=0).stop 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 msgr2=0x7f5634158360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 0x7f5634158360 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f562400d5d0 tx=0x7f562400da90 comp rx=0 tx=0).stop 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 shutdown_connections 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56140744b0 0x7f56140769c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56340d10b0 0x7f563415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5634070500 0x7f563409b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 --2- 172.21.15.146:0/521784997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f563406d560 0x7f5634158360 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.945+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 >> 172.21.15.146:0/521784997 conn(0x7f56340d1c10 msgr2=0x7f56341605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.946+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 shutdown_connections 2024-04-25T15:20:30.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:30.946+0000 7f5638ebd640 1 -- 172.21.15.146:0/521784997 wait complete. 2024-04-25T15:20:30.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:31.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.068+0000 7f3426670640 1 Processor -- start 2024-04-25T15:20:31.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.068+0000 7f3426670640 1 -- start start 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3426670640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3420070500 0x7f34200b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3426670640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34200d10b0 0x7f342015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3426670640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f342006d560 0x7f342006d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3426670640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f34200b0850 con 0x7f3420070500 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3426670640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f342006deb0 con 0x7f342006d560 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3426670640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f34200b0520 con 0x7f34200d10b0 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3420070500 0x7f34200b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3420070500 0x7f34200b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51206/0 (socket says 172.21.15.146:51206) 2024-04-25T15:20:31.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 -- 172.21.15.146:0/1285932729 learned_addr learned my addr 172.21.15.146:0/1285932729 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3424c0d640 1 --2- 172.21.15.146:0/1285932729 >> v2:172.21.15.177:3300/0 conn(0x7f342006d560 0x7f342006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341f7fe640 1 --2- 172.21.15.146:0/1285932729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34200d10b0 0x7f342015cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 -- 172.21.15.146:0/1285932729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34200d10b0 msgr2=0x7f342015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 --2- 172.21.15.146:0/1285932729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34200d10b0 0x7f342015cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 -- 172.21.15.146:0/1285932729 >> v2:172.21.15.177:3300/0 conn(0x7f342006d560 msgr2=0x7f342006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 --2- 172.21.15.146:0/1285932729 >> v2:172.21.15.177:3300/0 conn(0x7f342006d560 0x7f342006d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f341ffff640 1 -- 172.21.15.146:0/1285932729 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3420070d90 con 0x7f3420070500 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.070+0000 7f3424c0d640 1 --2- 172.21.15.146:0/1285932729 >> v2:172.21.15.177:3300/0 conn(0x7f342006d560 0x7f342006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.071+0000 7f341ffff640 1 --2- 172.21.15.146:0/1285932729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200b0ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3420073360 tx=0x7f341002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f05bfe7a8defb9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.071+0000 7f341effd640 1 -- 172.21.15.146:0/1285932729 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410037070 con 0x7f3420070500 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.071+0000 7f341effd640 1 -- 172.21.15.146:0/1285932729 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3410032440 con 0x7f3420070500 2024-04-25T15:20:31.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.071+0000 7f341effd640 1 -- 172.21.15.146:0/1285932729 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410032810 con 0x7f3420070500 2024-04-25T15:20:31.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 -- 172.21.15.146:0/1285932729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 msgr2=0x7f34200b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 --2- 172.21.15.146:0/1285932729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200b0ce0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3420073360 tx=0x7f341002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 -- 172.21.15.146:0/1285932729 shutdown_connections 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 --2- 172.21.15.146:0/1285932729 >> v2:172.21.15.177:3300/0 conn(0x7f342006d560 0x7f342006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 --2- 172.21.15.146:0/1285932729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34200d10b0 0x7f342015cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 --2- 172.21.15.146:0/1285932729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200b0ce0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 -- 172.21.15.146:0/1285932729 >> 172.21.15.146:0/1285932729 conn(0x7f34200d1c10 msgr2=0x7f34200733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 -- 172.21.15.146:0/1285932729 shutdown_connections 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.072+0000 7f3426670640 1 -- 172.21.15.146:0/1285932729 wait complete. 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.074+0000 7f3426670640 1 Processor -- start 2024-04-25T15:20:31.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.074+0000 7f3426670640 1 -- start start 2024-04-25T15:20:31.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3426670640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f342006d560 0x7f34200cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3426670640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3426670640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34200d10b0 0x7f3420150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3426670640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f342009f7d0 con 0x7f3420070500 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3426670640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f342009f650 con 0x7f34200d10b0 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3426670640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f34200b4610 con 0x7f342006d560 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51220/0 (socket says 172.21.15.146:51220) 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 -- 172.21.15.146:0/3137753059 learned_addr learned my addr 172.21.15.146:0/3137753059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341ffff640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f342006d560 0x7f34200cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3424c0d640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34200d10b0 0x7f3420150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 -- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f342006d560 msgr2=0x7f34200cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f342006d560 0x7f34200cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 -- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34200d10b0 msgr2=0x7f3420150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34200d10b0 0x7f3420150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341f7fe640 1 -- 172.21.15.146:0/3137753059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34201517f0 con 0x7f3420070500 2024-04-25T15:20:31.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f341ffff640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f342006d560 0x7f34200cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.075+0000 7f3424c0d640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34200d10b0 0x7f3420150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.076+0000 7f341f7fe640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200d0520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f341400d5f0 tx=0x7f341400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.076+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140125f0 con 0x7f3420070500 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.076+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3414019070 con 0x7f3420070500 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.076+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34200cd420 con 0x7f3420070500 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.076+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140129d0 con 0x7f3420070500 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.076+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34200ceaa0 con 0x7f3420070500 2024-04-25T15:20:31.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.077+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342006a0e0 con 0x7f3420070500 2024-04-25T15:20:31.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.077+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3414012b70 con 0x7f3420070500 2024-04-25T15:20:31.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.078+0000 7f341cff9640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f340c0744e0 0x7f340c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.078+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f34140958b0 con 0x7f3420070500 2024-04-25T15:20:31.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.078+0000 7f341ffff640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f340c0744e0 0x7f340c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.078+0000 7f341ffff640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f340c0744e0 0x7f340c0769f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3410034000 tx=0x7f341002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.083+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34201517f0 con 0x7f3420070500 2024-04-25T15:20:31.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.260+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f34200cddd0 con 0x7f3420070500 2024-04-25T15:20:31.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.260+0000 7f341cff9640 1 -- 172.21.15.146:0/3137753059 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f341405fc50 con 0x7f3420070500 2024-04-25T15:20:31.262 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:31.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f340c0744e0 msgr2=0x7f340c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f340c0744e0 0x7f340c0769f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3410034000 tx=0x7f341002e000 comp rx=0 tx=0).stop 2024-04-25T15:20:31.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 msgr2=0x7f34200d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200d0520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f341400d5f0 tx=0x7f341400dab0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 shutdown_connections 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f340c0744e0 0x7f340c0769f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34200d10b0 0x7f3420150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3420070500 0x7f34200d0520 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 --2- 172.21.15.146:0/3137753059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f342006d560 0x7f34200cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 >> 172.21.15.146:0/3137753059 conn(0x7f34200d1c10 msgr2=0x7f34200b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 shutdown_connections 2024-04-25T15:20:31.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.262+0000 7f3426670640 1 -- 172.21.15.146:0/3137753059 wait complete. 2024-04-25T15:20:31.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.381+0000 7fd467cd8640 1 Processor -- start 2024-04-25T15:20:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.382+0000 7fd467cd8640 1 -- start start 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd467cd8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd46006ee60 0x7fd4600b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd467cd8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4600b8eb0 con 0x7fd4601649d0 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd467cd8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd46008ce00 con 0x7fd460163eb0 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd467cd8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd460161840 con 0x7fd46006ee60 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd466275640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd4601649d0 0x7fd4600d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42830/0 (socket says 172.21.15.146:42830) 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd466275640 1 -- 172.21.15.146:0/1021452441 learned_addr learned my addr 172.21.15.146:0/1021452441 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd465a74640 1 --2- 172.21.15.146:0/1021452441 >> v2:172.21.15.177:3300/0 conn(0x7fd46006ee60 0x7fd4600b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3595766286 0 0) 0x7fd4600b8eb0 con 0x7fd4601649d0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.383+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd44c003520 con 0x7fd4601649d0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1467660960 0 0) 0x7fd44c003520 con 0x7fd4601649d0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 >> v2:172.21.15.177:3300/0 conn(0x7fd46006ee60 msgr2=0x7fd4600b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 --2- 172.21.15.146:0/1021452441 >> v2:172.21.15.177:3300/0 conn(0x7fd46006ee60 0x7fd4600b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 >> v1:172.21.15.177:6789/0 conn(0x7fd460163eb0 legacy=0x7fd4601642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46008d0e0 con 0x7fd4601649d0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fd4580019f0 con 0x7fd4601649d0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 >> v1:172.21.15.146:6789/0 conn(0x7fd4601649d0 legacy=0x7fd4600d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd44c0039b0 0x7fd44c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd44c004800 0x7fd44c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd44c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd460161840 con 0x7fd44c0051c0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c00c160 con 0x7fd44c004800 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd44c00c360 con 0x7fd44c0039b0 2024-04-25T15:20:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd466275640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd44c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd465273640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd44c004800 0x7fd44c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd465a74640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd44c0039b0 0x7fd44c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd466275640 1 -- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd44c0039b0 msgr2=0x7fd44c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd466275640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd44c0039b0 0x7fd44c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd466275640 1 -- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd44c004800 msgr2=0x7fd44c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd466275640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd44c004800 0x7fd44c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.384+0000 7fd466275640 1 -- 172.21.15.146:0/1021452441 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c003520 con 0x7fd44c0051c0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.385+0000 7fd466275640 1 -- 172.21.15.146:0/1021452441 reap_dead start 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.385+0000 7fd466275640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd44c00b9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd45800a830 tx=0x7fd44c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b838c7dcceb2a70a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.385+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45800a480 con 0x7fd44c0051c0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.385+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd45802f070 con 0x7fd44c0051c0 2024-04-25T15:20:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.385+0000 7fd464a72640 1 -- 172.21.15.146:0/1021452441 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45802bbc0 con 0x7fd44c0051c0 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 -- 172.21.15.146:0/1021452441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 msgr2=0x7fd44c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd44c00b9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd45800a830 tx=0x7fd44c005830 comp rx=0 tx=0).stop 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 -- 172.21.15.146:0/1021452441 shutdown_connections 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 --2- 172.21.15.146:0/1021452441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd44c00b9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 -- 172.21.15.146:0/1021452441 >> 172.21.15.146:0/1021452441 conn(0x7fd4600b52e0 msgr2=0x7fd4600b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 -- 172.21.15.146:0/1021452441 shutdown_connections 2024-04-25T15:20:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.386+0000 7fd467cd8640 1 -- 172.21.15.146:0/1021452441 wait complete. 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.388+0000 7fd467cd8640 1 Processor -- start 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.388+0000 7fd467cd8640 1 -- start start 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd467cd8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd4600b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd467cd8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd46015be90 0x7fd460162a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd467cd8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd460162f50 0x7fd4600b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd467cd8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd46008d730 con 0x7fd44c0051c0 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd467cd8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd46006f440 con 0x7fd46015be90 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd467cd8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd46008dbc0 con 0x7fd460162f50 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd4600b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd4600b2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51248/0 (socket says 172.21.15.146:51248) 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 -- 172.21.15.146:0/2695255697 learned_addr learned my addr 172.21.15.146:0/2695255697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd466275640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd460162f50 0x7fd4600b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465273640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd46015be90 0x7fd460162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 -- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd460162f50 msgr2=0x7fd4600b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd460162f50 0x7fd4600b6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 -- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd46015be90 msgr2=0x7fd460162a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd46015be90 0x7fd460162a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465a74640 1 -- 172.21.15.146:0/2695255697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd460163360 con 0x7fd44c0051c0 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd466275640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd460162f50 0x7fd4600b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.389+0000 7fd465273640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd46015be90 0x7fd460162a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.390+0000 7fd465a74640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd4600b2120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd45c011910 tx=0x7fd45c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.390+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c005ba0 con 0x7fd44c0051c0 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.390+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd45c005d40 con 0x7fd44c0051c0 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.390+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46015d910 con 0x7fd44c0051c0 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.390+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00fb00 con 0x7fd44c0051c0 2024-04-25T15:20:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.390+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46015cb50 con 0x7fd44c0051c0 2024-04-25T15:20:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.391+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd46006a0e0 con 0x7fd44c0051c0 2024-04-25T15:20:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.391+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd45c012040 con 0x7fd44c0051c0 2024-04-25T15:20:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.392+0000 7fd4567fc640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4440744b0 0x7fd4440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.392+0000 7fd465273640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4440744b0 0x7fd4440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.392+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fd45c095ca0 con 0x7fd44c0051c0 2024-04-25T15:20:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.393+0000 7fd465273640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4440744b0 0x7fd4440769c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd46015f5f0 tx=0x7fd450009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.397+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd460163360 con 0x7fd44c0051c0 2024-04-25T15:20:31.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.574+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4600b61f0 con 0x7fd44c0051c0 2024-04-25T15:20:31.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.575+0000 7fd4567fc640 1 -- 172.21.15.146:0/2695255697 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7fd45c018030 con 0x7fd44c0051c0 2024-04-25T15:20:31.577 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:31.578 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:31.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4440744b0 msgr2=0x7fd4440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4440744b0 0x7fd4440769c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd46015f5f0 tx=0x7fd450009040 comp rx=0 tx=0).stop 2024-04-25T15:20:31.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 msgr2=0x7fd4600b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd4600b2120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd45c011910 tx=0x7fd45c011dd0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 shutdown_connections 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4440744b0 0x7fd4440769c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd460162f50 0x7fd4600b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd46015be90 0x7fd460162a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 --2- 172.21.15.146:0/2695255697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd44c0051c0 0x7fd4600b2120 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 >> 172.21.15.146:0/2695255697 conn(0x7fd4600b52e0 msgr2=0x7fd46015e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 shutdown_connections 2024-04-25T15:20:31.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.577+0000 7fd467cd8640 1 -- 172.21.15.146:0/2695255697 wait complete. 2024-04-25T15:20:31.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:31.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:31.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.700+0000 7f3cfc61e640 1 Processor -- start 2024-04-25T15:20:31.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.700+0000 7f3cfc61e640 1 -- start start 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfc61e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cf415e070 0x7f3cf40cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfc61e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3cf40d2680 0x7f3cf40d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfc61e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf40d2fd0 con 0x7f3cf409bd80 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfc61e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf40ce920 con 0x7f3cf415e070 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfc61e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf40cec00 con 0x7f3cf40d2680 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfa3ba640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cf415e070 0x7f3cf40cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cf9bb9640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3cf409bd80 0x7f3cf40ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42844/0 (socket says 172.21.15.146:42844) 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfa3ba640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cf415e070 0x7f3cf40cf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51254/0 (socket says 172.21.15.146:51254) 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cf9bb9640 1 -- 172.21.15.146:0/307945235 learned_addr learned my addr 172.21.15.146:0/307945235 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfabbb640 1 --2- 172.21.15.146:0/307945235 >> v2:172.21.15.177:3300/0 conn(0x7f3cf40d2680 0x7f3cf40d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfa3ba640 1 -- 172.21.15.146:0/307945235 >> v2:172.21.15.177:3300/0 conn(0x7f3cf40d2680 msgr2=0x7f3cf40d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:31.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfa3ba640 1 --2- 172.21.15.146:0/307945235 >> v2:172.21.15.177:3300/0 conn(0x7f3cf40d2680 0x7f3cf40d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfa3ba640 1 -- 172.21.15.146:0/307945235 >> v1:172.21.15.146:6789/0 conn(0x7f3cf409bd80 legacy=0x7f3cf40ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cfa3ba640 1 -- 172.21.15.146:0/307945235 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf40ceee0 con 0x7f3cf415e070 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.702+0000 7f3cf93b8640 1 -- 172.21.15.146:0/307945235 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923513649 0 0) 0x7f3ce4001110 con 0x7f3cf409bd80 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.703+0000 7f3cfa3ba640 1 --2- 172.21.15.146:0/307945235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf40cf9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40b55f0 tx=0x7f3cf002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc756ee111b346c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.703+0000 7f3cf93b8640 1 -- 172.21.15.146:0/307945235 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0037070 con 0x7f3cf415e070 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.703+0000 7f3cf93b8640 1 -- 172.21.15.146:0/307945235 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf0032440 con 0x7f3cf415e070 2024-04-25T15:20:31.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.703+0000 7f3cf93b8640 1 -- 172.21.15.146:0/307945235 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0032810 con 0x7f3cf415e070 2024-04-25T15:20:31.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 -- 172.21.15.146:0/307945235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 msgr2=0x7f3cf40cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/307945235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf40cf9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40b55f0 tx=0x7f3cf002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 -- 172.21.15.146:0/307945235 shutdown_connections 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/307945235 >> v2:172.21.15.177:3300/0 conn(0x7f3cf40d2680 0x7f3cf40d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/307945235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf40cf9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 -- 172.21.15.146:0/307945235 >> 172.21.15.146:0/307945235 conn(0x7f3cf40b7400 msgr2=0x7f3cf40b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 -- 172.21.15.146:0/307945235 shutdown_connections 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.704+0000 7f3cfc61e640 1 -- 172.21.15.146:0/307945235 wait complete. 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.706+0000 7f3cfc61e640 1 Processor -- start 2024-04-25T15:20:31.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.706+0000 7f3cfc61e640 1 -- start start 2024-04-25T15:20:31.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfc61e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf409bd80 0x7f3cf415e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfc61e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf40d2680 0x7f3cf41f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfc61e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf4160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfc61e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf40cff00 con 0x7f3cf415e070 2024-04-25T15:20:31.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfc61e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf40cf4a0 con 0x7f3cf409bd80 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfc61e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf406cde0 con 0x7f3cf40d2680 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfabbb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfabbb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf4160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51260/0 (socket says 172.21.15.146:51260) 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfabbb640 1 -- 172.21.15.146:0/2049874089 learned_addr learned my addr 172.21.15.146:0/2049874089 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cfa3ba640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf409bd80 0x7f3cf415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.707+0000 7f3cf9bb9640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf40d2680 0x7f3cf41f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfabbb640 1 -- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf40d2680 msgr2=0x7f3cf41f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfabbb640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf40d2680 0x7f3cf41f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfabbb640 1 -- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf409bd80 msgr2=0x7f3cf415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfabbb640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf409bd80 0x7f3cf415e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfabbb640 1 -- 172.21.15.146:0/2049874089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf415ecb0 con 0x7f3cf415e070 2024-04-25T15:20:31.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cf9bb9640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf40d2680 0x7f3cf41f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:31.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfabbb640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf4160540 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009fd0 tx=0x7f3cec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec014420 con 0x7f3cf415e070 2024-04-25T15:20:31.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec019070 con 0x7f3cf415e070 2024-04-25T15:20:31.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec014800 con 0x7f3cf415e070 2024-04-25T15:20:31.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf40c7320 con 0x7f3cf415e070 2024-04-25T15:20:31.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.708+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf40c78e0 con 0x7f3cf415e070 2024-04-25T15:20:31.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.709+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf406a0e0 con 0x7f3cf415e070 2024-04-25T15:20:31.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.709+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3cec0149a0 con 0x7f3cf415e070 2024-04-25T15:20:31.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.710+0000 7f3ceaffd640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd80744e0 0x7f3cd80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:31.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.710+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f3cec095d10 con 0x7f3cf415e070 2024-04-25T15:20:31.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.710+0000 7f3cfa3ba640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd80744e0 0x7f3cd80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:31.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.710+0000 7f3cfa3ba640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd80744e0 0x7f3cd80769f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0034000 tx=0x7f3cf002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:31.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.715+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cec060140 con 0x7f3cf415e070 2024-04-25T15:20:31.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.888+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3cf415ef10 con 0x7f3cf415e070 2024-04-25T15:20:31.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.888+0000 7f3ceaffd640 1 -- 172.21.15.146:0/2049874089 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3cec05fa40 con 0x7f3cf415e070 2024-04-25T15:20:31.890 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd80744e0 msgr2=0x7f3cd80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd80744e0 0x7f3cd80769f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0034000 tx=0x7f3cf002e000 comp rx=0 tx=0).stop 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 msgr2=0x7f3cf4160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf4160540 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009fd0 tx=0x7f3cec00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 shutdown_connections 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd80744e0 0x7f3cd80769f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cf415e070 0x7f3cf4160540 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf40d2680 0x7f3cf41f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 --2- 172.21.15.146:0/2049874089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf409bd80 0x7f3cf415e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:31.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 >> 172.21.15.146:0/2049874089 conn(0x7f3cf40b7400 msgr2=0x7f3cf40b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:31.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 shutdown_connections 2024-04-25T15:20:31.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:31.890+0000 7f3cfc61e640 1 -- 172.21.15.146:0/2049874089 wait complete. 2024-04-25T15:20:31.904 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:32.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.015+0000 7fc96f651640 1 Processor -- start 2024-04-25T15:20:32.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.015+0000 7fc96f651640 1 -- start start 2024-04-25T15:20:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96f651640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc9680b2cd0 0x7fc9680cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96f651640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc9680d2440 0x7fc968071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96f651640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9680cf7d0 con 0x7fc9680d1920 2024-04-25T15:20:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96f651640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc96808da90 con 0x7fc9680d2440 2024-04-25T15:20:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96f651640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc9680d0070 con 0x7fc9680b2cd0 2024-04-25T15:20:32.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96dbee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc9680d2440 0x7fc968071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96cbec640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc9680d1920 0x7fc9680d1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42858/0 (socket says 172.21.15.146:42858) 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.016+0000 7fc96cbec640 1 -- 172.21.15.146:0/1897359181 learned_addr learned my addr 172.21.15.146:0/1897359181 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc96d3ed640 1 --2- 172.21.15.146:0/1897359181 >> v2:172.21.15.177:3300/0 conn(0x7fc9680b2cd0 0x7fc9680cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc96dbee640 1 -- 172.21.15.146:0/1897359181 >> v2:172.21.15.177:3300/0 conn(0x7fc9680b2cd0 msgr2=0x7fc9680cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc96dbee640 1 --2- 172.21.15.146:0/1897359181 >> v2:172.21.15.177:3300/0 conn(0x7fc9680b2cd0 0x7fc9680cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc95ffff640 1 -- 172.21.15.146:0/1897359181 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4620716 0 0) 0x7fc9680cf7d0 con 0x7fc9680d1920 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc96dbee640 1 -- 172.21.15.146:0/1897359181 >> v1:172.21.15.146:6789/0 conn(0x7fc9680d1920 legacy=0x7fc9680d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc96dbee640 1 -- 172.21.15.146:0/1897359181 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9680cfab0 con 0x7fc9680d2440 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc96dbee640 1 --2- 172.21.15.146:0/1897359181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d2440 0x7fc968071c70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc9680b6040 tx=0x7fc96002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d70aa39a9e8c11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc95ffff640 1 -- 172.21.15.146:0/1897359181 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960037070 con 0x7fc9680d2440 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc95ffff640 1 -- 172.21.15.146:0/1897359181 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc960032440 con 0x7fc9680d2440 2024-04-25T15:20:32.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.017+0000 7fc95ffff640 1 -- 172.21.15.146:0/1897359181 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96002ae60 con 0x7fc9680d2440 2024-04-25T15:20:32.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 -- 172.21.15.146:0/1897359181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d2440 msgr2=0x7fc968071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 --2- 172.21.15.146:0/1897359181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d2440 0x7fc968071c70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc9680b6040 tx=0x7fc96002a110 comp rx=0 tx=0).stop 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 -- 172.21.15.146:0/1897359181 shutdown_connections 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 --2- 172.21.15.146:0/1897359181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d2440 0x7fc968071c70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 --2- 172.21.15.146:0/1897359181 >> v2:172.21.15.177:3300/0 conn(0x7fc9680b2cd0 0x7fc9680cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 -- 172.21.15.146:0/1897359181 >> 172.21.15.146:0/1897359181 conn(0x7fc9680b52e0 msgr2=0x7fc9680b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 -- 172.21.15.146:0/1897359181 shutdown_connections 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.018+0000 7fc96f651640 1 -- 172.21.15.146:0/1897359181 wait complete. 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.020+0000 7fc96f651640 1 Processor -- start 2024-04-25T15:20:32.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.020+0000 7fc96f651640 1 -- start start 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96f651640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9680b2cd0 0x7fc9681f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96f651640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 0x7fc968160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96f651640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9680d2440 0x7fc9680cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96f651640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9680721b0 con 0x7fc9680d1920 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96f651640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc9680717e0 con 0x7fc9680d2440 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96f651640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc968072330 con 0x7fc9680b2cd0 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96cbec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 0x7fc968160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96cbec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 0x7fc968160430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51274/0 (socket says 172.21.15.146:51274) 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.021+0000 7fc96cbec640 1 -- 172.21.15.146:0/3525490074 learned_addr learned my addr 172.21.15.146:0/3525490074 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96dbee640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9680d2440 0x7fc9680cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96d3ed640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9680b2cd0 0x7fc9681f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96cbec640 1 -- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9680b2cd0 msgr2=0x7fc9681f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96cbec640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9680b2cd0 0x7fc9681f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96cbec640 1 -- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9680d2440 msgr2=0x7fc9680cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96cbec640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9680d2440 0x7fc9680cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96cbec640 1 -- 172.21.15.146:0/3525490074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9681f4f60 con 0x7fc9680d1920 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96cbec640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 0x7fc968160430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc95800c410 tx=0x7fc95800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9580074e0 con 0x7fc9680d1920 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9580047e0 con 0x7fc9680d1920 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9681f5160 con 0x7fc9680d1920 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958004bd0 con 0x7fc9680d1920 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.022+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9681f5420 con 0x7fc9680d1920 2024-04-25T15:20:32.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.023+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc96806a0e0 con 0x7fc9680d1920 2024-04-25T15:20:32.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.024+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc958011070 con 0x7fc9680d1920 2024-04-25T15:20:32.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.024+0000 7fc95dffb640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc94c0744b0 0x7fc94c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.024+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fc958095210 con 0x7fc9680d1920 2024-04-25T15:20:32.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.024+0000 7fc96d3ed640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc94c0744b0 0x7fc94c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.025+0000 7fc96d3ed640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc94c0744b0 0x7fc94c0769c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc9640070b0 tx=0x7fc964007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.029+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9681f4f60 con 0x7fc9680d1920 2024-04-25T15:20:32.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.202+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc968161550 con 0x7fc9680d1920 2024-04-25T15:20:32.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.203+0000 7fc95dffb640 1 -- 172.21.15.146:0/3525490074 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc95805f5b0 con 0x7fc9680d1920 2024-04-25T15:20:32.205 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc94c0744b0 msgr2=0x7fc94c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc94c0744b0 0x7fc94c0769c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc9640070b0 tx=0x7fc964007040 comp rx=0 tx=0).stop 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 msgr2=0x7fc968160430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 0x7fc968160430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc95800c410 tx=0x7fc95800c8d0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 shutdown_connections 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc94c0744b0 0x7fc94c0769c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9680d2440 0x7fc9680cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9680d1920 0x7fc968160430 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 --2- 172.21.15.146:0/3525490074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9680b2cd0 0x7fc9681f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 >> 172.21.15.146:0/3525490074 conn(0x7fc9680b52e0 msgr2=0x7fc9680d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 shutdown_connections 2024-04-25T15:20:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.205+0000 7fc96f651640 1 -- 172.21.15.146:0/3525490074 wait complete. 2024-04-25T15:20:32.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:32.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.327+0000 7fd663642640 1 Processor -- start 2024-04-25T15:20:32.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.327+0000 7fd663642640 1 -- start start 2024-04-25T15:20:32.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd663642640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd65c0b2cd0 0x7fd65c0cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd663642640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd65c0d1920 0x7fd65c0d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd663642640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65c0cf7d0 con 0x7fd65c0d2270 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd663642640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd65c0d0070 con 0x7fd65c0d1920 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd663642640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd65c0717e0 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd65c0d1920 0x7fd65c0d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd65c0d1920 0x7fd65c0d1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51282/0 (socket says 172.21.15.146:51282) 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 -- 172.21.15.146:0/3960186971 learned_addr learned my addr 172.21.15.146:0/3960186971 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd6613de640 1 --2- 172.21.15.146:0/3960186971 >> v2:172.21.15.177:3300/0 conn(0x7fd65c0b2cd0 0x7fd65c0cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 -- 172.21.15.146:0/3960186971 >> v2:172.21.15.177:3300/0 conn(0x7fd65c0b2cd0 msgr2=0x7fd65c0cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 --2- 172.21.15.146:0/3960186971 >> v2:172.21.15.177:3300/0 conn(0x7fd65c0b2cd0 0x7fd65c0cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 -- 172.21.15.146:0/3960186971 >> v1:172.21.15.177:6789/0 conn(0x7fd65c0d2270 legacy=0x7fd65c08ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.329+0000 7fd660bdd640 1 -- 172.21.15.146:0/3960186971 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c0cfab0 con 0x7fd65c0d1920 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.330+0000 7fd660bdd640 1 --2- 172.21.15.146:0/3960186971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0d1920 0x7fd65c0d1d30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0b6040 tx=0x7fd64c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4d581cc075d556 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.330+0000 7fd653fff640 1 -- 172.21.15.146:0/3960186971 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c037070 con 0x7fd65c0d1920 2024-04-25T15:20:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.330+0000 7fd653fff640 1 -- 172.21.15.146:0/3960186971 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64c032440 con 0x7fd65c0d1920 2024-04-25T15:20:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.330+0000 7fd653fff640 1 -- 172.21.15.146:0/3960186971 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c032810 con 0x7fd65c0d1920 2024-04-25T15:20:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 -- 172.21.15.146:0/3960186971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0d1920 msgr2=0x7fd65c0d1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 --2- 172.21.15.146:0/3960186971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0d1920 0x7fd65c0d1d30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0b6040 tx=0x7fd64c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 -- 172.21.15.146:0/3960186971 shutdown_connections 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 --2- 172.21.15.146:0/3960186971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0d1920 0x7fd65c0d1d30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 --2- 172.21.15.146:0/3960186971 >> v2:172.21.15.177:3300/0 conn(0x7fd65c0b2cd0 0x7fd65c0cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 -- 172.21.15.146:0/3960186971 >> 172.21.15.146:0/3960186971 conn(0x7fd65c0b52e0 msgr2=0x7fd65c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 -- 172.21.15.146:0/3960186971 shutdown_connections 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.331+0000 7fd663642640 1 -- 172.21.15.146:0/3960186971 wait complete. 2024-04-25T15:20:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.333+0000 7fd663642640 1 Processor -- start 2024-04-25T15:20:32.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.333+0000 7fd663642640 1 -- start start 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd663642640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 0x7fd65c0ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd663642640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd65c0d1920 0x7fd65c161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd663642640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd65c0d2270 0x7fd65c150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd663642640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c072c80 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd663642640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c0d2680 con 0x7fd65c0d1920 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd663642640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c072e00 con 0x7fd65c0d2270 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 0x7fd65c0ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 0x7fd65c0ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51290/0 (socket says 172.21.15.146:51290) 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 -- 172.21.15.146:0/4001554809 learned_addr learned my addr 172.21.15.146:0/4001554809 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd660bdd640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd65c0d1920 0x7fd65c161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd661bdf640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd65c0d2270 0x7fd65c150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 -- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd65c0d2270 msgr2=0x7fd65c150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd65c0d2270 0x7fd65c150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 -- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd65c0d1920 msgr2=0x7fd65c161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd65c0d1920 0x7fd65c161230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.334+0000 7fd6613de640 1 -- 172.21.15.146:0/4001554809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c09f880 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.335+0000 7fd6613de640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 0x7fd65c0ca8a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d830 tx=0x7fd65800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.335+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800be10 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.335+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd658005990 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.335+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658005b80 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.335+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c09fb20 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.335+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c0a00e0 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.336+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65c06a0e0 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.336+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd65801b020 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.337+0000 7fd651ffb640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd6400744b0 0x7fd6400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.337+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fd6580953d0 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.337+0000 7fd660bdd640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd6400744b0 0x7fd6400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.337+0000 7fd660bdd640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd6400744b0 0x7fd6400769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd64c034000 tx=0x7fd64c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.341+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd65805f770 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.517+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd65c0b1ce0 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.517+0000 7fd651ffb640 1 -- 172.21.15.146:0/4001554809 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fd65805f070 con 0x7fd65c0b2cd0 2024-04-25T15:20:32.519 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:32.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd6400744b0 msgr2=0x7fd6400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd6400744b0 0x7fd6400769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd64c034000 tx=0x7fd64c02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:32.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 msgr2=0x7fd65c0ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 0x7fd65c0ca8a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d830 tx=0x7fd65800dcf0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 shutdown_connections 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd6400744b0 0x7fd6400769c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd65c0d2270 0x7fd65c150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd65c0d1920 0x7fd65c161230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 --2- 172.21.15.146:0/4001554809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd65c0b2cd0 0x7fd65c0ca8a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 >> 172.21.15.146:0/4001554809 conn(0x7fd65c0b52e0 msgr2=0x7fd65c0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 shutdown_connections 2024-04-25T15:20:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.519+0000 7fd663642640 1 -- 172.21.15.146:0/4001554809 wait complete. 2024-04-25T15:20:32.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.641+0000 7f43cdab6640 1 Processor -- start 2024-04-25T15:20:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.641+0000 7f43cdab6640 1 -- start start 2024-04-25T15:20:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43cdab6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 0x7f43c80d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43cdab6640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 0x7f43c80cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43cdab6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c80d2260 con 0x7f43c80d3300 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43cdab6640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f43c80d2540 con 0x7f43c8071d30 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43cdab6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f43c80d2820 con 0x7f43c80b2cd0 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c7fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 0x7f43c80cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c77fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 0x7f43c80d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c6ffd640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f43c80d3300 0x7f43c8071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:47196/0 (socket says 172.21.15.146:47196) 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c6ffd640 1 -- 172.21.15.146:0/2246480327 learned_addr learned my addr 172.21.15.146:0/2246480327 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c77fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 0x7f43c80d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:41182/0 (socket says 172.21.15.146:41182) 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c7fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 0x7f43c80cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48192/0 (socket says 172.21.15.146:48192) 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c77fe640 1 -- 172.21.15.146:0/2246480327 >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 msgr2=0x7f43c80cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c77fe640 1 --2- 172.21.15.146:0/2246480327 >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 0x7f43c80cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.643+0000 7f43c77fe640 1 -- 172.21.15.146:0/2246480327 >> v1:172.21.15.177:6789/0 conn(0x7f43c80d3300 legacy=0x7f43c8071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c67fc640 1 -- 172.21.15.146:0/2246480327 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856647731 0 0) 0x7f43c80d2260 con 0x7f43c80d3300 2024-04-25T15:20:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c77fe640 1 -- 172.21.15.146:0/2246480327 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c80d2b00 con 0x7f43c80b2cd0 2024-04-25T15:20:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c7fff640 1 --2- 172.21.15.146:0/2246480327 >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 0x7f43c80cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:32.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c77fe640 1 --2- 172.21.15.146:0/2246480327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 0x7f43c80d1c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009790 tx=0x7f43b802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b116273b867d8c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c67fc640 1 -- 172.21.15.146:0/2246480327 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8037070 con 0x7f43c80b2cd0 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c67fc640 1 -- 172.21.15.146:0/2246480327 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b8032440 con 0x7f43c80b2cd0 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.644+0000 7f43c67fc640 1 -- 172.21.15.146:0/2246480327 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b802ae60 con 0x7f43c80b2cd0 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 -- 172.21.15.146:0/2246480327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 msgr2=0x7f43c80d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 --2- 172.21.15.146:0/2246480327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 0x7f43c80d1c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009790 tx=0x7f43b802a110 comp rx=0 tx=0).stop 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 -- 172.21.15.146:0/2246480327 shutdown_connections 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 --2- 172.21.15.146:0/2246480327 >> v2:172.21.15.177:3300/0 conn(0x7f43c8071d30 0x7f43c80cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 --2- 172.21.15.146:0/2246480327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b2cd0 0x7f43c80d1c80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 -- 172.21.15.146:0/2246480327 >> 172.21.15.146:0/2246480327 conn(0x7f43c80b52e0 msgr2=0x7f43c80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 -- 172.21.15.146:0/2246480327 shutdown_connections 2024-04-25T15:20:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.645+0000 7f43cdab6640 1 -- 172.21.15.146:0/2246480327 wait complete. 2024-04-25T15:20:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.647+0000 7f43cdab6640 1 Processor -- start 2024-04-25T15:20:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.647+0000 7f43cdab6640 1 -- start start 2024-04-25T15:20:32.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.648+0000 7f43cdab6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 0x7f43c80b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.648+0000 7f43cdab6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c80d3300 0x7f43c80cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.648+0000 7f43cdab6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b79f0 0x7f43c806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.648+0000 7f43cdab6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f43c808dbf0 con 0x7f43c80b2cd0 2024-04-25T15:20:32.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.648+0000 7f43cdab6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43c80cfb70 con 0x7f43c80d3300 2024-04-25T15:20:32.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.648+0000 7f43cdab6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f43c808dd70 con 0x7f43c80b79f0 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 0x7f43c80b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 0x7f43c80b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51306/0 (socket says 172.21.15.146:51306) 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 -- 172.21.15.146:0/44716270 learned_addr learned my addr 172.21.15.146:0/44716270 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c6ffd640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c80d3300 0x7f43c80cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c7fff640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b79f0 0x7f43c806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 -- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b79f0 msgr2=0x7f43c806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b79f0 0x7f43c806ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 -- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c80d3300 msgr2=0x7f43c80cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c80d3300 0x7f43c80cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 -- 172.21.15.146:0/44716270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c80b8340 con 0x7f43c80b2cd0 2024-04-25T15:20:32.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43c77fe640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 0x7f43c80b6f90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f43b8035040 tx=0x7f43b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8042020 con 0x7f43c80b2cd0 2024-04-25T15:20:32.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b8038040 con 0x7f43c80b2cd0 2024-04-25T15:20:32.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c80b64b0 con 0x7f43c80b2cd0 2024-04-25T15:20:32.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8037050 con 0x7f43c80b2cd0 2024-04-25T15:20:32.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.649+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c80b6a70 con 0x7f43c80b2cd0 2024-04-25T15:20:32.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.650+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c806a0e0 con 0x7f43c80b2cd0 2024-04-25T15:20:32.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.651+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f43b8005990 con 0x7f43c80b2cd0 2024-04-25T15:20:32.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.651+0000 7f43affff640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43b40744e0 0x7f43b40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.652+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f43b80b5aa0 con 0x7f43c80b2cd0 2024-04-25T15:20:32.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.652+0000 7f43c6ffd640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43b40744e0 0x7f43b40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.652+0000 7f43c6ffd640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43b40744e0 0x7f43b40769f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f43c80686c0 tx=0x7f43bc007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.656+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43b807f890 con 0x7f43c80b2cd0 2024-04-25T15:20:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.832+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f43c815faf0 con 0x7f43c80b2cd0 2024-04-25T15:20:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.832+0000 7f43affff640 1 -- 172.21.15.146:0/44716270 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f43b8083b90 con 0x7f43c80b2cd0 2024-04-25T15:20:32.834 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43b40744e0 msgr2=0x7f43b40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43b40744e0 0x7f43b40769f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f43c80686c0 tx=0x7f43bc007de0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 msgr2=0x7f43c80b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 0x7f43c80b6f90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f43b8035040 tx=0x7f43b80046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 shutdown_connections 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43b40744e0 0x7f43b40769f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43c80b79f0 0x7f43c806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43c80d3300 0x7f43c80cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 --2- 172.21.15.146:0/44716270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43c80b2cd0 0x7f43c80b6f90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 >> 172.21.15.146:0/44716270 conn(0x7f43c80b52e0 msgr2=0x7f43c8071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 shutdown_connections 2024-04-25T15:20:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.834+0000 7f43cdab6640 1 -- 172.21.15.146:0/44716270 wait complete. 2024-04-25T15:20:32.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:32.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.954+0000 7f0e4c3ee640 1 Processor -- start 2024-04-25T15:20:32.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.954+0000 7f0e4c3ee640 1 -- start start 2024-04-25T15:20:32.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4c3ee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e44160e00 0x7f0e4415c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4c3ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4c3ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e440b10b0 con 0x7f0e4406cde0 2024-04-25T15:20:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4c3ee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0e4415c9d0 con 0x7f0e44160e00 2024-04-25T15:20:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4c3ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0e441600d0 con 0x7f0e440ccfb0 2024-04-25T15:20:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4a18a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e44160e00 0x7f0e4415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e49989640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0e4406cde0 0x7f0e4406d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42868/0 (socket says 172.21.15.146:42868) 2024-04-25T15:20:32.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e49989640 1 -- 172.21.15.146:0/296303193 learned_addr learned my addr 172.21.15.146:0/296303193 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:32.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4a18a640 1 -- 172.21.15.146:0/296303193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 msgr2=0x7f0e440cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:32.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4a18a640 1 --2- 172.21.15.146:0/296303193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4a18a640 1 -- 172.21.15.146:0/296303193 >> v1:172.21.15.146:6789/0 conn(0x7f0e4406cde0 legacy=0x7f0e4406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e4a18a640 1 -- 172.21.15.146:0/296303193 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e440cdc50 con 0x7f0e44160e00 2024-04-25T15:20:32.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.956+0000 7f0e49188640 1 -- 172.21.15.146:0/296303193 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 229713666 0 0) 0x7f0e440b10b0 con 0x7f0e4406cde0 2024-04-25T15:20:32.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.957+0000 7f0e4a18a640 1 --2- 172.21.15.146:0/296303193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e44160e00 0x7f0e4415c490 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e400097a0 tx=0x7f0e4002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f43f870b1fdcf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.957+0000 7f0e49188640 1 -- 172.21.15.146:0/296303193 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40037070 con 0x7f0e44160e00 2024-04-25T15:20:32.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.957+0000 7f0e49188640 1 -- 172.21.15.146:0/296303193 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e40032440 con 0x7f0e44160e00 2024-04-25T15:20:32.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.957+0000 7f0e49188640 1 -- 172.21.15.146:0/296303193 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40032810 con 0x7f0e44160e00 2024-04-25T15:20:32.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/296303193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e44160e00 msgr2=0x7f0e4415c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/296303193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e44160e00 0x7f0e4415c490 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e400097a0 tx=0x7f0e4002a180 comp rx=0 tx=0).stop 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/296303193 shutdown_connections 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/296303193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/296303193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e44160e00 0x7f0e4415c490 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/296303193 >> 172.21.15.146:0/296303193 conn(0x7f0e440ce0c0 msgr2=0x7f0e440ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/296303193 shutdown_connections 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.958+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/296303193 wait complete. 2024-04-25T15:20:32.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.959+0000 7f0e4c3ee640 1 Processor -- start 2024-04-25T15:20:32.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.960+0000 7f0e4c3ee640 1 -- start start 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4c3ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 0x7f0e440ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4c3ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4c3ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e44158d10 0x7f0e44155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4c3ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0e441608e0 con 0x7f0e4406cde0 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4c3ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0e44160600 con 0x7f0e44158d10 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4c3ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0e440cd900 con 0x7f0e440ccfb0 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 0x7f0e440ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 0x7f0e440ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51330/0 (socket says 172.21.15.146:51330) 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 -- 172.21.15.146:0/1795630271 learned_addr learned my addr 172.21.15.146:0/1795630271 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e49989640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a98b640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e44158d10 0x7f0e44155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 -- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 msgr2=0x7f0e440b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 -- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e44158d10 msgr2=0x7f0e44155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e44158d10 0x7f0e44155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a18a640 1 -- 172.21.15.146:0/1795630271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e440b0ed0 con 0x7f0e4406cde0 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e49989640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.961+0000 7f0e4a98b640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e44158d10 0x7f0e44155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.962+0000 7f0e4a18a640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 0x7f0e440ae5f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e40000c00 tx=0x7f0e400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.962+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40041070 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.962+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e4003c930 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.962+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e440aee40 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.962+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40037040 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.962+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e4409d220 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.963+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e440af360 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.963+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0e40044070 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.964+0000 7f0e3affd640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0e280744b0 0x7f0e280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.964+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f0e400b61d0 con 0x7f0e4406cde0 2024-04-25T15:20:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.964+0000 7f0e49989640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0e280744b0 0x7f0e280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:32.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.964+0000 7f0e49989640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0e280744b0 0x7f0e280769c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e440b20f0 tx=0x7f0e34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:32.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:32.968+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e440af360 con 0x7f0e4406cde0 2024-04-25T15:20:33.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.144+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e4406a0e0 con 0x7f0e4406cde0 2024-04-25T15:20:33.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.145+0000 7f0e3affd640 1 -- 172.21.15.146:0/1795630271 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f0e40080570 con 0x7f0e4406cde0 2024-04-25T15:20:33.147 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:33.147 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:33.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0e280744b0 msgr2=0x7f0e280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0e280744b0 0x7f0e280769c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e440b20f0 tx=0x7f0e34009040 comp rx=0 tx=0).stop 2024-04-25T15:20:33.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 msgr2=0x7f0e440ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 0x7f0e440ae5f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e40000c00 tx=0x7f0e400046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 shutdown_connections 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0e280744b0 0x7f0e280769c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e44158d10 0x7f0e44155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e440ccfb0 0x7f0e440b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 --2- 172.21.15.146:0/1795630271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e4406cde0 0x7f0e440ae5f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 >> 172.21.15.146:0/1795630271 conn(0x7f0e440ce0c0 msgr2=0x7f0e4415eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 shutdown_connections 2024-04-25T15:20:33.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.147+0000 7f0e4c3ee640 1 -- 172.21.15.146:0/1795630271 wait complete. 2024-04-25T15:20:33.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:33.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:33.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.270+0000 7f15b3934640 1 Processor -- start 2024-04-25T15:20:33.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.271+0000 7f15b3934640 1 -- start start 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b3934640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15ac0b2cd0 0x7f15ac0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b3934640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac0d3300 0x7f15ac071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b3934640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f15ac071b60 0x7f15ac08cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b3934640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f15ac071f70 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b3934640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f15ac0d2260 con 0x7f15ac071b60 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b3934640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f15ac0d2540 con 0x7f15ac0d3300 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15ac0b2cd0 0x7f15ac0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15ac0b2cd0 0x7f15ac0d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51332/0 (socket says 172.21.15.146:51332) 2024-04-25T15:20:33.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 -- 172.21.15.146:0/1261787922 learned_addr learned my addr 172.21.15.146:0/1261787922 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b0ecf640 1 --2- 172.21.15.146:0/1261787922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac0d3300 0x7f15ac071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b1ed1640 1 --2- 172.21.15.146:0/1261787922 >> v2:172.21.15.177:3300/0 conn(0x7f15ac071b60 0x7f15ac08cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 -- 172.21.15.146:0/1261787922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac0d3300 msgr2=0x7f15ac071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 --2- 172.21.15.146:0/1261787922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac0d3300 0x7f15ac071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 -- 172.21.15.146:0/1261787922 >> v2:172.21.15.177:3300/0 conn(0x7f15ac071b60 msgr2=0x7f15ac08cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 --2- 172.21.15.146:0/1261787922 >> v2:172.21.15.177:3300/0 conn(0x7f15ac071b60 0x7f15ac08cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.272+0000 7f15b16d0640 1 -- 172.21.15.146:0/1261787922 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac0d2820 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.273+0000 7f15b16d0640 1 --2- 172.21.15.146:0/1261787922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0d1c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0744d0 tx=0x7f15a802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1e155b3ba16303 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.273+0000 7f15a3fff640 1 -- 172.21.15.146:0/1261787922 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8036070 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.273+0000 7f15a3fff640 1 -- 172.21.15.146:0/1261787922 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a8031600 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.273+0000 7f15a3fff640 1 -- 172.21.15.146:0/1261787922 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a80319d0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 -- 172.21.15.146:0/1261787922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 msgr2=0x7f15ac0d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 --2- 172.21.15.146:0/1261787922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0d1c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0744d0 tx=0x7f15a802a660 comp rx=0 tx=0).stop 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 -- 172.21.15.146:0/1261787922 shutdown_connections 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 --2- 172.21.15.146:0/1261787922 >> v2:172.21.15.177:3300/0 conn(0x7f15ac071b60 0x7f15ac08cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 --2- 172.21.15.146:0/1261787922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac0d3300 0x7f15ac071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 --2- 172.21.15.146:0/1261787922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0d1c80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 -- 172.21.15.146:0/1261787922 >> 172.21.15.146:0/1261787922 conn(0x7f15ac0b52e0 msgr2=0x7f15ac0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 -- 172.21.15.146:0/1261787922 shutdown_connections 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.274+0000 7f15b3934640 1 -- 172.21.15.146:0/1261787922 wait complete. 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.276+0000 7f15b3934640 1 Processor -- start 2024-04-25T15:20:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.276+0000 7f15b3934640 1 -- start start 2024-04-25T15:20:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b3934640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac071b60 0x7f15ac0b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b3934640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b3934640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac0d3300 0x7f15ac0cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b3934640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac0cfb70 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b3934640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac0d2de0 con 0x7f15ac0d3300 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b3934640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f15ac08dbf0 con 0x7f15ac071b60 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b0ecf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b0ecf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51336/0 (socket says 172.21.15.146:51336) 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b0ecf640 1 -- 172.21.15.146:0/2170100139 learned_addr learned my addr 172.21.15.146:0/2170100139 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b1ed1640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac0d3300 0x7f15ac0cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.277+0000 7f15b16d0640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac071b60 0x7f15ac0b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b0ecf640 1 -- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac071b60 msgr2=0x7f15ac0b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b0ecf640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac071b60 0x7f15ac0b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b0ecf640 1 -- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac0d3300 msgr2=0x7f15ac0cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b0ecf640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac0d3300 0x7f15ac0cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b0ecf640 1 -- 172.21.15.146:0/2170100139 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac160260 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b16d0640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac071b60 0x7f15ac0b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b1ed1640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac0d3300 0x7f15ac0cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:33.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b0ecf640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0cb8f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f159c00d620 tx=0x7f159c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c00b620 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f159c019070 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c00ba00 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac1610f0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.278+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac1616b0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.279+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ac06a0e0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.280+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f159c00bba0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.280+0000 7f15a1ffb640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15900744e0 0x7f15900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.280+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f159c095a40 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.280+0000 7f15b16d0640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15900744e0 0x7f15900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.281+0000 7f15b16d0640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15900744e0 0x7f15900769f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15a8033000 tx=0x7f15a8009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.285+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f159c05fde0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.457+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15ac160e10 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.458+0000 7f15a1ffb640 1 -- 172.21.15.146:0/2170100139 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f159c05f6e0 con 0x7f15ac0b2cd0 2024-04-25T15:20:33.460 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15900744e0 msgr2=0x7f15900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15900744e0 0x7f15900769f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15a8033000 tx=0x7f15a8009000 comp rx=0 tx=0).stop 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 msgr2=0x7f15ac0cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0cb8f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f159c00d620 tx=0x7f159c00dae0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 shutdown_connections 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f15900744e0 0x7f15900769f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac0d3300 0x7f15ac0cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15ac0b2cd0 0x7f15ac0cb8f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 --2- 172.21.15.146:0/2170100139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15ac071b60 0x7f15ac0b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 >> 172.21.15.146:0/2170100139 conn(0x7f15ac0b52e0 msgr2=0x7f15ac0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:33.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 shutdown_connections 2024-04-25T15:20:33.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.460+0000 7f15b3934640 1 -- 172.21.15.146:0/2170100139 wait complete. 2024-04-25T15:20:33.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:33.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.582+0000 7f6aecf38640 1 Processor -- start 2024-04-25T15:20:33.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.583+0000 7f6aecf38640 1 -- start start 2024-04-25T15:20:33.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6aecf38640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6ae8160b00 0x7f6ae8160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6aecf38640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae806f7b0 con 0x7f6ae815c440 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6aecf38640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae809b700 con 0x7f6ae806f2a0 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6aecf38640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f6ae80b1f30 con 0x7f6ae8160b00 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6ae6d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6ae815c440 0x7f6ae815c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42882/0 (socket says 172.21.15.146:42882) 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6ae6d9d640 1 -- 172.21.15.146:0/3984483360 learned_addr learned my addr 172.21.15.146:0/3984483360 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6ae5d9b640 1 --2- 172.21.15.146:0/3984483360 >> v2:172.21.15.177:3300/0 conn(0x7f6ae8160b00 0x7f6ae8160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.584+0000 7f6ae559a640 1 -- 172.21.15.146:0/3984483360 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219598425 0 0) 0x7f6ae806f7b0 con 0x7f6ae815c440 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.585+0000 7f6ae559a640 1 -- 172.21.15.146:0/3984483360 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ad0003520 con 0x7f6ae815c440 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.585+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/3984483360 >> v1:172.21.15.177:6789/0 conn(0x7f6ae806f2a0 legacy=0x7f6ae80cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.585+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/3984483360 >> v1:172.21.15.146:6789/0 conn(0x7f6ae815c440 legacy=0x7f6ae815c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.585+0000 7f6ae559a640 1 -- 172.21.15.146:0/3984483360 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 116356721 0 0) 0x7f6ad0003520 con 0x7f6ae815c440 2024-04-25T15:20:33.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.585+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/3984483360 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae809b9e0 con 0x7f6ae8160b00 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.585+0000 7f6ae5d9b640 1 --2- 172.21.15.146:0/3984483360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae8160f10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0097a0 tx=0x7f6adc02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3612405b783627b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6ae559a640 1 -- 172.21.15.146:0/3984483360 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc037070 con 0x7f6ae8160b00 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6ae559a640 1 -- 172.21.15.146:0/3984483360 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc02ac50 con 0x7f6ae8160b00 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6ae559a640 1 -- 172.21.15.146:0/3984483360 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0001670 con 0x7f6ae8160b00 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6aecf38640 1 -- 172.21.15.146:0/3984483360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 msgr2=0x7f6ae8160f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6aecf38640 1 --2- 172.21.15.146:0/3984483360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae8160f10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0097a0 tx=0x7f6adc02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6aecf38640 1 -- 172.21.15.146:0/3984483360 shutdown_connections 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6aecf38640 1 --2- 172.21.15.146:0/3984483360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae8160f10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.586+0000 7f6aecf38640 1 -- 172.21.15.146:0/3984483360 >> 172.21.15.146:0/3984483360 conn(0x7f6ae80b7490 msgr2=0x7f6ae80b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:33.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.587+0000 7f6aecf38640 1 -- 172.21.15.146:0/3984483360 shutdown_connections 2024-04-25T15:20:33.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.587+0000 7f6aecf38640 1 -- 172.21.15.146:0/3984483360 wait complete. 2024-04-25T15:20:33.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.588+0000 7f6aecf38640 1 Processor -- start 2024-04-25T15:20:33.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.588+0000 7f6aecf38640 1 -- start start 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6aecf38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ae806f2a0 0x7f6ae80d1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6aecf38640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 0x7f6ae80b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6aecf38640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae80b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6aecf38640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae80ce420 con 0x7f6ae815c440 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6aecf38640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae80ce2a0 con 0x7f6ae8160b00 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6aecf38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae80ce5a0 con 0x7f6ae806f2a0 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 0x7f6ae80b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 0x7f6ae80b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51352/0 (socket says 172.21.15.146:51352) 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/2128429628 learned_addr learned my addr 172.21.15.146:0/2128429628 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae6d9d640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae80b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ae806f2a0 msgr2=0x7f6ae80d1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae659c640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ae806f2a0 0x7f6ae80d1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ae806f2a0 0x7f6ae80d1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 msgr2=0x7f6ae80b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae80b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 -- 172.21.15.146:0/2128429628 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae80c4eb0 con 0x7f6ae815c440 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae6d9d640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae80b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ae5d9b640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 0x7f6ae80b0880 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0098c0 tx=0x7f6adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.590+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc041020 con 0x7f6ae815c440 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.591+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc0044c0 con 0x7f6ae815c440 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.591+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae80c5190 con 0x7f6ae815c440 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.591+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc037040 con 0x7f6ae815c440 2024-04-25T15:20:33.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.591+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae80c5750 con 0x7f6ae815c440 2024-04-25T15:20:33.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.592+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae806a0e0 con 0x7f6ae815c440 2024-04-25T15:20:33.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.592+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6adc044020 con 0x7f6ae815c440 2024-04-25T15:20:33.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.593+0000 7f6ad6ffd640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6ac80744b0 0x7f6ac80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.593+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f6adc0b5a00 con 0x7f6ae815c440 2024-04-25T15:20:33.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.593+0000 7f6ae659c640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6ac80744b0 0x7f6ac80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.593+0000 7f6ae659c640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6ac80744b0 0x7f6ac80769c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8007fd0 tx=0x7f6ad8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.597+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6adc04b030 con 0x7f6ae815c440 2024-04-25T15:20:33.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.774+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6ae80c5f30 con 0x7f6ae815c440 2024-04-25T15:20:33.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.775+0000 7f6ad6ffd640 1 -- 172.21.15.146:0/2128429628 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f6adc07fe30 con 0x7f6ae815c440 2024-04-25T15:20:33.777 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:33.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6ac80744b0 msgr2=0x7f6ac80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6ac80744b0 0x7f6ac80769c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8007fd0 tx=0x7f6ad8009040 comp rx=0 tx=0).stop 2024-04-25T15:20:33.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 msgr2=0x7f6ae80b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 0x7f6ae80b0880 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0098c0 tx=0x7f6adc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 shutdown_connections 2024-04-25T15:20:33.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6ac80744b0 0x7f6ac80769c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ae8160b00 0x7f6ae80b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ae815c440 0x7f6ae80b0880 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 --2- 172.21.15.146:0/2128429628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ae806f2a0 0x7f6ae80d1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 >> 172.21.15.146:0/2128429628 conn(0x7f6ae80b7490 msgr2=0x7f6ae80b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:33.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 shutdown_connections 2024-04-25T15:20:33.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.777+0000 7f6aecf38640 1 -- 172.21.15.146:0/2128429628 wait complete. 2024-04-25T15:20:33.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:33.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.899+0000 7fb35d7f3640 1 Processor -- start 2024-04-25T15:20:33.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.899+0000 7fb35d7f3640 1 -- start start 2024-04-25T15:20:33.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.900+0000 7fb35d7f3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb3580d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.900+0000 7fb35d7f3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb358071d30 0x7fb3580cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.900+0000 7fb35d7f3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3580d2260 con 0x7fb3580d3300 2024-04-25T15:20:33.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.900+0000 7fb35d7f3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb3580d2540 con 0x7fb358071d30 2024-04-25T15:20:33.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.900+0000 7fb35d7f3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb3580d2820 con 0x7fb3580b2cd0 2024-04-25T15:20:33.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb358071d30 0x7fb3580cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb358071d30 0x7fb3580cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51360/0 (socket says 172.21.15.146:51360) 2024-04-25T15:20:33.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 -- 172.21.15.146:0/2252455723 learned_addr learned my addr 172.21.15.146:0/2252455723 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb356ffd640 1 --2- 172.21.15.146:0/2252455723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb3580d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 -- 172.21.15.146:0/2252455723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 msgr2=0x7fb3580d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 --2- 172.21.15.146:0/2252455723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb3580d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 -- 172.21.15.146:0/2252455723 >> v1:172.21.15.146:6789/0 conn(0x7fb3580d3300 legacy=0x7fb358071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb355ffb640 1 -- 172.21.15.146:0/2252455723 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617775170 0 0) 0x7fb3580d2260 con 0x7fb3580d3300 2024-04-25T15:20:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.901+0000 7fb3577fe640 1 -- 172.21.15.146:0/2252455723 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3580d2b00 con 0x7fb358071d30 2024-04-25T15:20:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.902+0000 7fb3577fe640 1 --2- 172.21.15.146:0/2252455723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb358071d30 0x7fb3580cde40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb350009870 tx=0x7fb35002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb770b659b414af8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.902+0000 7fb355ffb640 1 -- 172.21.15.146:0/2252455723 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350037070 con 0x7fb358071d30 2024-04-25T15:20:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.902+0000 7fb355ffb640 1 -- 172.21.15.146:0/2252455723 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb350032440 con 0x7fb358071d30 2024-04-25T15:20:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.902+0000 7fb355ffb640 1 -- 172.21.15.146:0/2252455723 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350032810 con 0x7fb358071d30 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.902+0000 7fb35d7f3640 1 -- 172.21.15.146:0/2252455723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb358071d30 msgr2=0x7fb3580cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.902+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/2252455723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb358071d30 0x7fb3580cde40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb350009870 tx=0x7fb35002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.903+0000 7fb35d7f3640 1 -- 172.21.15.146:0/2252455723 shutdown_connections 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.903+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/2252455723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb358071d30 0x7fb3580cde40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.903+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/2252455723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb3580d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.903+0000 7fb35d7f3640 1 -- 172.21.15.146:0/2252455723 >> 172.21.15.146:0/2252455723 conn(0x7fb3580b52e0 msgr2=0x7fb3580b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.903+0000 7fb35d7f3640 1 -- 172.21.15.146:0/2252455723 shutdown_connections 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.903+0000 7fb35d7f3640 1 -- 172.21.15.146:0/2252455723 wait complete. 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.904+0000 7fb35d7f3640 1 Processor -- start 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.904+0000 7fb35d7f3640 1 -- start start 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb35d7f3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb358158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb35d7f3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 0x7fb3580cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb35d7f3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35806ef20 0x7fb35806f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb35d7f3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb35808dbf0 con 0x7fb3580d3300 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb35d7f3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580cfb70 con 0x7fb35806ef20 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb35d7f3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb35808dd70 con 0x7fb3580b2cd0 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 0x7fb3580cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 0x7fb3580cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51366/0 (socket says 172.21.15.146:51366) 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 -- 172.21.15.146:0/3857419378 learned_addr learned my addr 172.21.15.146:0/3857419378 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 -- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 msgr2=0x7fb358158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3577fe640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35806ef20 0x7fb35806f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb356ffd640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb358158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb358158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 -- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35806ef20 msgr2=0x7fb35806f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35806ef20 0x7fb35806f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 -- 172.21.15.146:0/3857419378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb358159310 con 0x7fb3580d3300 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.906+0000 7fb3567fc640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 0x7fb3580cbc00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c440 tx=0x7fb34c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.907+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c017070 con 0x7fb3580d3300 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.907+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c0035f0 con 0x7fb3580d3300 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.907+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35806f870 con 0x7fb3580d3300 2024-04-25T15:20:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.907+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0039e0 con 0x7fb3580d3300 2024-04-25T15:20:33.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.907+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3580ca940 con 0x7fb3580d3300 2024-04-25T15:20:33.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.907+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35806a0e0 con 0x7fb3580d3300 2024-04-25T15:20:33.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.908+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb34c01a070 con 0x7fb3580d3300 2024-04-25T15:20:33.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.909+0000 7fb33f7fe640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3440744e0 0x7fb3440769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:33.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.909+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fb34c0948c0 con 0x7fb3580d3300 2024-04-25T15:20:33.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.909+0000 7fb356ffd640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3440744e0 0x7fb3440769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:33.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.909+0000 7fb356ffd640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3440744e0 0x7fb3440769f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb3580b9490 tx=0x7fb34800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:33.913+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb35806f870 con 0x7fb3580d3300 2024-04-25T15:20:34.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.090+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb358068cf0 con 0x7fb3580d3300 2024-04-25T15:20:34.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.090+0000 7fb33f7fe640 1 -- 172.21.15.146:0/3857419378 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb34c092e20 con 0x7fb3580d3300 2024-04-25T15:20:34.092 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:34.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3440744e0 msgr2=0x7fb3440769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3440744e0 0x7fb3440769f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb3580b9490 tx=0x7fb34800b040 comp rx=0 tx=0).stop 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 msgr2=0x7fb3580cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 0x7fb3580cbc00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c440 tx=0x7fb34c00c900 comp rx=0 tx=0).stop 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 shutdown_connections 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3440744e0 0x7fb3440769f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35806ef20 0x7fb35806f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3580d3300 0x7fb3580cbc00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 --2- 172.21.15.146:0/3857419378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3580b2cd0 0x7fb358158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.092+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 >> 172.21.15.146:0/3857419378 conn(0x7fb3580b52e0 msgr2=0x7fb3580b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.093+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 shutdown_connections 2024-04-25T15:20:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.093+0000 7fb35d7f3640 1 -- 172.21.15.146:0/3857419378 wait complete. 2024-04-25T15:20:34.105 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:34.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.215+0000 7f68ad630640 1 Processor -- start 2024-04-25T15:20:34.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.215+0000 7f68ad630640 1 -- start start 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.216+0000 7f68ad630640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68a8163c00 0x7f68a806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68ad630640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a806dea0 con 0x7f68a80d0fb0 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68ad630640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a80b5230 con 0x7f68a80bb450 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68ad630640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f68a806c8c0 con 0x7f68a8163c00 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68a8163c00 0x7f68a806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a77fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f68a80d0fb0 0x7f68a80b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42902/0 (socket says 172.21.15.146:42902) 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a77fe640 1 -- 172.21.15.146:0/4186292533 learned_addr learned my addr 172.21.15.146:0/4186292533 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a6ffd640 1 -- 172.21.15.146:0/4186292533 >> v1:172.21.15.177:6789/0 conn(0x7f68a80bb450 legacy=0x7f68a80d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a6ffd640 1 -- 172.21.15.146:0/4186292533 >> v1:172.21.15.146:6789/0 conn(0x7f68a80d0fb0 legacy=0x7f68a80b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a5ffb640 1 -- 172.21.15.146:0/4186292533 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89470324 0 0) 0x7f68a806dea0 con 0x7f68a80d0fb0 2024-04-25T15:20:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.217+0000 7f68a6ffd640 1 -- 172.21.15.146:0/4186292533 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a806cba0 con 0x7f68a8163c00 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.218+0000 7f68a6ffd640 1 --2- 172.21.15.146:0/4186292533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8163c00 0x7f68a806d960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f68980097a0 tx=0x7f689802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735d701829606d84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.218+0000 7f68a5ffb640 1 -- 172.21.15.146:0/4186292533 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898037070 con 0x7f68a8163c00 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.218+0000 7f68a5ffb640 1 -- 172.21.15.146:0/4186292533 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6898032440 con 0x7f68a8163c00 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.218+0000 7f68a5ffb640 1 -- 172.21.15.146:0/4186292533 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898032810 con 0x7f68a8163c00 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.218+0000 7f68ad630640 1 -- 172.21.15.146:0/4186292533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8163c00 msgr2=0x7f68a806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.218+0000 7f68ad630640 1 --2- 172.21.15.146:0/4186292533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8163c00 0x7f68a806d960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f68980097a0 tx=0x7f689802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.219+0000 7f68ad630640 1 -- 172.21.15.146:0/4186292533 shutdown_connections 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.219+0000 7f68ad630640 1 --2- 172.21.15.146:0/4186292533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8163c00 0x7f68a806d960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.219+0000 7f68ad630640 1 -- 172.21.15.146:0/4186292533 >> 172.21.15.146:0/4186292533 conn(0x7f68a80b7970 msgr2=0x7f68a80b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.219+0000 7f68ad630640 1 -- 172.21.15.146:0/4186292533 shutdown_connections 2024-04-25T15:20:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.219+0000 7f68ad630640 1 -- 172.21.15.146:0/4186292533 wait complete. 2024-04-25T15:20:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.220+0000 7f68ad630640 1 Processor -- start 2024-04-25T15:20:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.221+0000 7f68ad630640 1 -- start start 2024-04-25T15:20:34.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 0x7f68a80c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a80d0fb0 0x7f68a815de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8163c00 0x7f68a815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f68a806c550 con 0x7f68a80bb450 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f68a806d160 con 0x7f68a80d0fb0 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f68a80d13c0 con 0x7f68a8163c00 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 0x7f68a80c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 0x7f68a80c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51378/0 (socket says 172.21.15.146:51378) 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 -- 172.21.15.146:0/285320920 learned_addr learned my addr 172.21.15.146:0/285320920 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 -- 172.21.15.146:0/285320920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8163c00 msgr2=0x7f68a815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8163c00 0x7f68a815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 -- 172.21.15.146:0/285320920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a80d0fb0 msgr2=0x7f68a815de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a80d0fb0 0x7f68a815de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 -- 172.21.15.146:0/285320920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a80c9bc0 con 0x7f68a80bb450 2024-04-25T15:20:34.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68a6ffd640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 0x7f68a80c9180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f68980097a0 tx=0x7f68980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898037050 con 0x7f68a80bb450 2024-04-25T15:20:34.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6898038040 con 0x7f68a80bb450 2024-04-25T15:20:34.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a80c9ea0 con 0x7f68a80bb450 2024-04-25T15:20:34.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898004490 con 0x7f68a80bb450 2024-04-25T15:20:34.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.222+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a815c4f0 con 0x7f68a80bb450 2024-04-25T15:20:34.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.223+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a80d4580 con 0x7f68a80bb450 2024-04-25T15:20:34.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.224+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6898030900 con 0x7f68a80bb450 2024-04-25T15:20:34.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.224+0000 7f688f7fe640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68940744e0 0x7f68940769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.225+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f68980b5fb0 con 0x7f68a80bb450 2024-04-25T15:20:34.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.225+0000 7f68a67fc640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68940744e0 0x7f68940769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.225+0000 7f68a67fc640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68940744e0 0x7f68940769f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f689c000f60 tx=0x7f689c0079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.229+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68980803e0 con 0x7f68a80bb450 2024-04-25T15:20:34.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.402+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f68a806a0e0 con 0x7f68a80bb450 2024-04-25T15:20:34.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.403+0000 7f688f7fe640 1 -- 172.21.15.146:0/285320920 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f68980883d0 con 0x7f68a80bb450 2024-04-25T15:20:34.405 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68940744e0 msgr2=0x7f68940769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68940744e0 0x7f68940769f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f689c000f60 tx=0x7f689c0079b0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 msgr2=0x7f68a80c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 0x7f68a80c9180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f68980097a0 tx=0x7f68980046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 shutdown_connections 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68940744e0 0x7f68940769f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8163c00 0x7f68a815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a80d0fb0 0x7f68a815de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 --2- 172.21.15.146:0/285320920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a80bb450 0x7f68a80c9180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 >> 172.21.15.146:0/285320920 conn(0x7f68a80b7970 msgr2=0x7f68a80d4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 shutdown_connections 2024-04-25T15:20:34.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.405+0000 7f68ad630640 1 -- 172.21.15.146:0/285320920 wait complete. 2024-04-25T15:20:34.561 INFO:tasks.cephfs_test_runner:Starting test: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) That a file with ceph.quiesce.block is not quiesced. 2024-04-25T15:20:34.562 INFO:tasks.cephfs_test_runner:test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-25T15:20:34.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.669+0000 7f211eed9640 1 Processor -- start 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.670+0000 7f211eed9640 1 -- start start 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f211eed9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2118153ca0 0x7f211815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f211eed9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21180cf420 con 0x7f21180d0ac0 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f211eed9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21180cc270 con 0x7f21180cbc90 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f211eed9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f21180cfd10 con 0x7f2118153ca0 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f2117fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2118153ca0 0x7f211815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f211d476640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f21180d0ac0 0x7f21180d0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42916/0 (socket says 172.21.15.146:42916) 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f2117fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2118153ca0 0x7f211815dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51380/0 (socket says 172.21.15.146:51380) 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f211d476640 1 -- 172.21.15.146:0/3813935878 learned_addr learned my addr 172.21.15.146:0/3813935878 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f2117fff640 1 -- 172.21.15.146:0/3813935878 >> v1:172.21.15.177:6789/0 conn(0x7f21180cbc90 legacy=0x7f21180d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f2117fff640 1 -- 172.21.15.146:0/3813935878 >> v1:172.21.15.146:6789/0 conn(0x7f21180d0ac0 legacy=0x7f21180d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f2117fff640 1 -- 172.21.15.146:0/3813935878 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21180cf6c0 con 0x7f2118153ca0 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.671+0000 7f21177fe640 1 -- 172.21.15.146:0/3813935878 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 53642294 0 0) 0x7f21180cf420 con 0x7f21180d0ac0 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.672+0000 7f2117fff640 1 --2- 172.21.15.146:0/3813935878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f211815dc60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21040097a0 tx=0x7f210402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0beb03dab512cc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.672+0000 7f21177fe640 1 -- 172.21.15.146:0/3813935878 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104037070 con 0x7f2118153ca0 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.672+0000 7f21177fe640 1 -- 172.21.15.146:0/3813935878 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2104032440 con 0x7f2118153ca0 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.672+0000 7f21177fe640 1 -- 172.21.15.146:0/3813935878 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104032810 con 0x7f2118153ca0 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 -- 172.21.15.146:0/3813935878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 msgr2=0x7f211815dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 --2- 172.21.15.146:0/3813935878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f211815dc60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21040097a0 tx=0x7f210402a230 comp rx=0 tx=0).stop 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 -- 172.21.15.146:0/3813935878 shutdown_connections 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 --2- 172.21.15.146:0/3813935878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f211815dc60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 -- 172.21.15.146:0/3813935878 >> 172.21.15.146:0/3813935878 conn(0x7f21180ce0c0 msgr2=0x7f21180ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 -- 172.21.15.146:0/3813935878 shutdown_connections 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.673+0000 7f211eed9640 1 -- 172.21.15.146:0/3813935878 wait complete. 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.675+0000 7f211eed9640 1 Processor -- start 2024-04-25T15:20:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.675+0000 7f211eed9640 1 -- start start 2024-04-25T15:20:34.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211eed9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21180cbc90 0x7f211814d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211eed9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21180d0ac0 0x7f2118150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211eed9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f2118150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211eed9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f21181540b0 con 0x7f2118153ca0 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211eed9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f211815e1a0 con 0x7f21180cbc90 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211eed9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2118070390 con 0x7f21180d0ac0 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211d476640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f2118150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211d476640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f2118150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51386/0 (socket says 172.21.15.146:51386) 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.676+0000 7f211d476640 1 -- 172.21.15.146:0/4042733234 learned_addr learned my addr 172.21.15.146:0/4042733234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f2117fff640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21180d0ac0 0x7f2118150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211cc75640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21180cbc90 0x7f211814d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211d476640 1 -- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21180d0ac0 msgr2=0x7f2118150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211d476640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21180d0ac0 0x7f2118150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211d476640 1 -- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21180cbc90 msgr2=0x7f211814d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211d476640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21180cbc90 0x7f211814d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211d476640 1 -- 172.21.15.146:0/4042733234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2118157f70 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211cc75640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21180cbc90 0x7f211814d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211d476640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f2118150650 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f211000c6d0 tx=0x7f211000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110018070 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2110004d60 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2118158250 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211000a820 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.677+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f211815a530 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.678+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f211806a0e0 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.679+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f211001b070 con 0x7f2118153ca0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.679+0000 7f21157fa640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20fc0744b0 0x7f20fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.679+0000 7f211cc75640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20fc0744b0 0x7f20fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.679+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f2110094a60 con 0x7f2118153ca0 2024-04-25T15:20:34.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.680+0000 7f211cc75640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20fc0744b0 0x7f20fc0769c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f210c000f60 tx=0x7f210c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.684+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f211005ee90 con 0x7f2118153ca0 2024-04-25T15:20:34.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.864+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21180711c0 con 0x7f2118153ca0 2024-04-25T15:20:34.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.865+0000 7f21157fa640 1 -- 172.21.15.146:0/4042733234 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f211005e790 con 0x7f2118153ca0 2024-04-25T15:20:34.867 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:34.867 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20fc0744b0 msgr2=0x7f20fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20fc0744b0 0x7f20fc0769c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f210c000f60 tx=0x7f210c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 msgr2=0x7f2118150650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f2118150650 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f211000c6d0 tx=0x7f211000cb90 comp rx=0 tx=0).stop 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 shutdown_connections 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20fc0744b0 0x7f20fc0769c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2118153ca0 0x7f2118150650 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21180d0ac0 0x7f2118150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 --2- 172.21.15.146:0/4042733234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21180cbc90 0x7f211814d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 >> 172.21.15.146:0/4042733234 conn(0x7f21180ce0c0 msgr2=0x7f211815e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:34.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 shutdown_connections 2024-04-25T15:20:34.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.867+0000 7f211eed9640 1 -- 172.21.15.146:0/4042733234 wait complete. 2024-04-25T15:20:34.880 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:34.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.990+0000 7f8091ff3640 1 Processor -- start 2024-04-25T15:20:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.990+0000 7f8091ff3640 1 -- start start 2024-04-25T15:20:34.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f8091ff3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f808c0b4ca0 0x7f808c0b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f8091ff3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:34.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f8091ff3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f808c08d590 con 0x7f808c08dd50 2024-04-25T15:20:34.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f8091ff3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f808c08d8c0 con 0x7f808c0b4ca0 2024-04-25T15:20:34.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f8091ff3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f808c0d27d0 con 0x7f808c0ce6f0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f808affd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f808c0b4ca0 0x7f808c0b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f808b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f808c08dd50 0x7f808c0d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42926/0 (socket says 172.21.15.146:42926) 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.991+0000 7f808b7fe640 1 -- 172.21.15.146:0/521017863 learned_addr learned my addr 172.21.15.146:0/521017863 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808bfff640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808affd640 1 -- 172.21.15.146:0/521017863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 msgr2=0x7f808c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808affd640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808affd640 1 -- 172.21.15.146:0/521017863 >> v1:172.21.15.146:6789/0 conn(0x7f808c08dd50 legacy=0x7f808c0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808affd640 1 -- 172.21.15.146:0/521017863 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808c0b48b0 con 0x7f808c0b4ca0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808a7fc640 1 -- 172.21.15.146:0/521017863 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350954280 0 0) 0x7f808c08d590 con 0x7f808c08dd50 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808bfff640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.992+0000 7f808affd640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8f30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f808002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531899b39bc4a438 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f808a7fc640 1 -- 172.21.15.146:0/521017863 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080037070 con 0x7f808c0b4ca0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f808a7fc640 1 -- 172.21.15.146:0/521017863 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8080032440 con 0x7f808c0b4ca0 2024-04-25T15:20:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f808a7fc640 1 -- 172.21.15.146:0/521017863 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080032810 con 0x7f808c0b4ca0 2024-04-25T15:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 -- 172.21.15.146:0/521017863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c0b4ca0 msgr2=0x7f808c0b8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8f30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f80800097a0 tx=0x7f808002a180 comp rx=0 tx=0).stop 2024-04-25T15:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 -- 172.21.15.146:0/521017863 shutdown_connections 2024-04-25T15:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 --2- 172.21.15.146:0/521017863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8f30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 -- 172.21.15.146:0/521017863 >> 172.21.15.146:0/521017863 conn(0x7f808c0b5430 msgr2=0x7f808c0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 -- 172.21.15.146:0/521017863 shutdown_connections 2024-04-25T15:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.993+0000 7f8091ff3640 1 -- 172.21.15.146:0/521017863 wait complete. 2024-04-25T15:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.995+0000 7f8091ff3640 1 Processor -- start 2024-04-25T15:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.995+0000 7f8091ff3640 1 -- start start 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0b87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 0x7f808c1f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f808c0d24c0 con 0x7f808c1d3390 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f808c0d2340 con 0x7f808c0b4ca0 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f808c0cf040 con 0x7f808c0ce6f0 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 0x7f808c1f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 0x7f808c1f78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51408/0 (socket says 172.21.15.146:51408) 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 -- 172.21.15.146:0/1568247259 learned_addr learned my addr 172.21.15.146:0/1568247259 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808affd640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0b87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808b7fe640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 -- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 msgr2=0x7f808c0b87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0b87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 -- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f808c0b4ca0 msgr2=0x7f808c0b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 -- 172.21.15.146:0/1568247259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808c09e730 con 0x7f808c1d3390 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808b7fe640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808affd640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f808bfff640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 0x7f808c1f78c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f808400cdb0 tx=0x7f808400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808400f680 con 0x7f808c1d3390 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808400f840 con 0x7f808c1d3390 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808400fc30 con 0x7f808c1d3390 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.997+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808c0b74b0 con 0x7f808c1d3390 2024-04-25T15:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.998+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f808c0b7a70 con 0x7f808c1d3390 2024-04-25T15:20:35.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.999+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808c0734d0 con 0x7f808c1d3390 2024-04-25T15:20:35.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:34.999+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f808400e3e0 con 0x7f808c1d3390 2024-04-25T15:20:35.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.000+0000 7f8073fff640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80780744b0 0x7f80780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.000+0000 7f808b7fe640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80780744b0 0x7f80780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.000+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f80840963f0 con 0x7f808c1d3390 2024-04-25T15:20:35.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.000+0000 7f808b7fe640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80780744b0 0x7f80780769c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f807c000f60 tx=0x7f807c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.005+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f808c0734d0 con 0x7f808c1d3390 2024-04-25T15:20:35.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.177+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f808c0d3300 con 0x7f808c1d3390 2024-04-25T15:20:35.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.177+0000 7f8073fff640 1 -- 172.21.15.146:0/1568247259 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f808c0d3300 con 0x7f808c1d3390 2024-04-25T15:20:35.179 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:35.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80780744b0 msgr2=0x7f80780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80780744b0 0x7f80780769c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f807c000f60 tx=0x7f807c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:35.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 msgr2=0x7f808c1f78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 0x7f808c1f78c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f808400cdb0 tx=0x7f808400a650 comp rx=0 tx=0).stop 2024-04-25T15:20:35.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 shutdown_connections 2024-04-25T15:20:35.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f80780744b0 0x7f80780769c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c1d3390 0x7f808c1f78c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f808c0ce6f0 0x7f808c0b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 --2- 172.21.15.146:0/1568247259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f808c0b4ca0 0x7f808c0b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 >> 172.21.15.146:0/1568247259 conn(0x7f808c0b5430 msgr2=0x7f808c0d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:35.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 shutdown_connections 2024-04-25T15:20:35.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.179+0000 7f8091ff3640 1 -- 172.21.15.146:0/1568247259 wait complete. 2024-04-25T15:20:35.192 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:35.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.300+0000 7f51aac92640 1 Processor -- start 2024-04-25T15:20:35.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.300+0000 7f51aac92640 1 -- start start 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51aac92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51a40bb450 0x7f51a40d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51aac92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a40b5230 con 0x7f51a40d0de0 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51aac92640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a406c8c0 con 0x7f51a4163c00 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51aac92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f51a406cba0 con 0x7f51a40bb450 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51a40bb450 0x7f51a40d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a922f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f51a40d0de0 0x7f51a40b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42938/0 (socket says 172.21.15.146:42938) 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a922f640 1 -- 172.21.15.146:0/2573674854 learned_addr learned my addr 172.21.15.146:0/2573674854 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a3fff640 1 -- 172.21.15.146:0/2573674854 >> v1:172.21.15.177:6789/0 conn(0x7f51a4163c00 legacy=0x7f51a406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a3fff640 1 -- 172.21.15.146:0/2573674854 >> v1:172.21.15.146:6789/0 conn(0x7f51a40d0de0 legacy=0x7f51a40b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a37fe640 1 -- 172.21.15.146:0/2573674854 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2414645749 0 0) 0x7f51a40b5230 con 0x7f51a40d0de0 2024-04-25T15:20:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.302+0000 7f51a3fff640 1 -- 172.21.15.146:0/2573674854 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a406ce80 con 0x7f51a40bb450 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.303+0000 7f51a3fff640 1 --2- 172.21.15.146:0/2573674854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a40bb450 0x7f51a40d08a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f51980097d0 tx=0x7f519802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64da9a2b04164120 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.303+0000 7f51a37fe640 1 -- 172.21.15.146:0/2573674854 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198037070 con 0x7f51a40bb450 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.303+0000 7f51a37fe640 1 -- 172.21.15.146:0/2573674854 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5198032440 con 0x7f51a40bb450 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.303+0000 7f51a37fe640 1 -- 172.21.15.146:0/2573674854 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198032810 con 0x7f51a40bb450 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 -- 172.21.15.146:0/2573674854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a40bb450 msgr2=0x7f51a40d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 --2- 172.21.15.146:0/2573674854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a40bb450 0x7f51a40d08a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f51980097d0 tx=0x7f519802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 -- 172.21.15.146:0/2573674854 shutdown_connections 2024-04-25T15:20:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 --2- 172.21.15.146:0/2573674854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a40bb450 0x7f51a40d08a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 -- 172.21.15.146:0/2573674854 >> 172.21.15.146:0/2573674854 conn(0x7f51a40b7970 msgr2=0x7f51a40b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 -- 172.21.15.146:0/2573674854 shutdown_connections 2024-04-25T15:20:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.304+0000 7f51aac92640 1 -- 172.21.15.146:0/2573674854 wait complete. 2024-04-25T15:20:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.305+0000 7f51aac92640 1 Processor -- start 2024-04-25T15:20:35.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.305+0000 7f51aac92640 1 -- start start 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51aac92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a40bb450 0x7f51a415c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51aac92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51a40d0de0 0x7f51a415cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51aac92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 0x7f51a415d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51aac92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51a40bb860 con 0x7f51a4163c00 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51aac92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51a406c550 con 0x7f51a40bb450 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51aac92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f51a40bb9e0 con 0x7f51a40d0de0 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 0x7f51a415d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 0x7f51a415d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51430/0 (socket says 172.21.15.146:51430) 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 -- 172.21.15.146:0/1606947351 learned_addr learned my addr 172.21.15.146:0/1606947351 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 -- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51a40d0de0 msgr2=0x7f51a415cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51a40d0de0 0x7f51a415cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 -- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a40bb450 msgr2=0x7f51a415c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a8a2e640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a40bb450 0x7f51a415c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a40bb450 0x7f51a415c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.307+0000 7f51a922f640 1 -- 172.21.15.146:0/1606947351 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a40cf1f0 con 0x7f51a4163c00 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.308+0000 7f51a922f640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 0x7f51a415d4f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f519c00c700 tx=0x7f519c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.308+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c018070 con 0x7f51a4163c00 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.308+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f519c00ce10 con 0x7f51a4163c00 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.308+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a4157ed0 con 0x7f51a4163c00 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.308+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c00a880 con 0x7f51a4163c00 2024-04-25T15:20:35.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.308+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a415cb60 con 0x7f51a4163c00 2024-04-25T15:20:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.309+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a415f380 con 0x7f51a4163c00 2024-04-25T15:20:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.309+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f519c01b070 con 0x7f51a4163c00 2024-04-25T15:20:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.310+0000 7f51a17fa640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51900744b0 0x7f51900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.310+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f519c094ad0 con 0x7f51a4163c00 2024-04-25T15:20:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.310+0000 7f51a8a2e640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51900744b0 0x7f51900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.310+0000 7f51a8a2e640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51900744b0 0x7f51900769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5194000f60 tx=0x7f5194009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.315+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f519c05ef00 con 0x7f51a4163c00 2024-04-25T15:20:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.489+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f51a406a0e0 con 0x7f51a4163c00 2024-04-25T15:20:35.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.490+0000 7f51a17fa640 1 -- 172.21.15.146:0/1606947351 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f519c0650d0 con 0x7f51a4163c00 2024-04-25T15:20:35.492 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51900744b0 msgr2=0x7f51900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51900744b0 0x7f51900769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5194000f60 tx=0x7f5194009040 comp rx=0 tx=0).stop 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 msgr2=0x7f51a415d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 0x7f51a415d4f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f519c00c700 tx=0x7f519c00cbc0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 shutdown_connections 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51900744b0 0x7f51900769c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51a4163c00 0x7f51a415d4f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51a40d0de0 0x7f51a415cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 --2- 172.21.15.146:0/1606947351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a40bb450 0x7f51a415c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 >> 172.21.15.146:0/1606947351 conn(0x7f51a40b7970 msgr2=0x7f51a40ba670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:35.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 shutdown_connections 2024-04-25T15:20:35.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.492+0000 7f51aac92640 1 -- 172.21.15.146:0/1606947351 wait complete. 2024-04-25T15:20:35.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:35.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.614+0000 7fac86b9d640 1 Processor -- start 2024-04-25T15:20:35.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.614+0000 7fac86b9d640 1 -- start start 2024-04-25T15:20:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac86b9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac800d0af0 0x7fac800b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac86b9d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac80163d80 con 0x7fac801638a0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac86b9d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac80164060 con 0x7fac800d1060 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac86b9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fac80164340 con 0x7fac800d0af0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac85b9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fac801638a0 0x7fac800d03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42950/0 (socket says 172.21.15.146:42950) 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac85b9b640 1 -- 172.21.15.146:0/2604931694 learned_addr learned my addr 172.21.15.146:0/2604931694 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac8539a640 1 --2- 172.21.15.146:0/2604931694 >> v2:172.21.15.177:3300/0 conn(0x7fac800d0af0 0x7fac800b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 63600212 0 0) 0x7fac80163d80 con 0x7fac801638a0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac6c003520 con 0x7fac801638a0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2807885377 0 0) 0x7fac6c003520 con 0x7fac801638a0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 >> v2:172.21.15.177:3300/0 conn(0x7fac800d0af0 msgr2=0x7fac800b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 --2- 172.21.15.146:0/2604931694 >> v2:172.21.15.177:3300/0 conn(0x7fac800d0af0 0x7fac800b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 >> v1:172.21.15.177:6789/0 conn(0x7fac800d1060 legacy=0x7fac8015f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac80164620 con 0x7fac801638a0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fac7c0019f0 con 0x7fac801638a0 2024-04-25T15:20:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 >> v1:172.21.15.146:6789/0 conn(0x7fac801638a0 legacy=0x7fac800d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac6c0039b0 0x7fac6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac6c004800 0x7fac6c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0051c0 0x7fac6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac80164340 con 0x7fac6c0039b0 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c00c160 con 0x7fac6c004800 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fac6c00c360 con 0x7fac6c0051c0 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.616+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac6c0039b0 0x7fac6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac8539a640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac6c004800 0x7fac6c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac8639c640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0051c0 0x7fac6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac85b9b640 1 -- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0051c0 msgr2=0x7fac6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0051c0 0x7fac6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac85b9b640 1 -- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac6c004800 msgr2=0x7fac6c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac6c004800 0x7fac6c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac85b9b640 1 -- 172.21.15.146:0/2604931694 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c003520 con 0x7fac6c0039b0 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac8539a640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac6c004800 0x7fac6c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac8639c640 1 -- 172.21.15.146:0/2604931694 reap_dead start 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.617+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac6c0039b0 0x7fac6c003e20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00a830 tx=0x7fac6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40665d107f37a0ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00a480 con 0x7fac6c0039b0 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac7c02f070 con 0x7fac6c0039b0 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac84b99640 1 -- 172.21.15.146:0/2604931694 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c02bc40 con 0x7fac6c0039b0 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 -- 172.21.15.146:0/2604931694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac6c0039b0 msgr2=0x7fac6c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac6c0039b0 0x7fac6c003e20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fac7c00a830 tx=0x7fac6c005830 comp rx=0 tx=0).stop 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 -- 172.21.15.146:0/2604931694 shutdown_connections 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2604931694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac6c0039b0 0x7fac6c003e20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 -- 172.21.15.146:0/2604931694 >> 172.21.15.146:0/2604931694 conn(0x7fac800d2ba0 msgr2=0x7fac800b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 -- 172.21.15.146:0/2604931694 shutdown_connections 2024-04-25T15:20:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.618+0000 7fac86b9d640 1 -- 172.21.15.146:0/2604931694 wait complete. 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.620+0000 7fac86b9d640 1 Processor -- start 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.620+0000 7fac86b9d640 1 -- start start 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac86b9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0039b0 0x7fac80157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac86b9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 0x7fac800b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac86b9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac800b6b50 0x7fac801f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac86b9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac800b4fa0 con 0x7fac800cf1a0 2024-04-25T15:20:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac86b9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac800b4e20 con 0x7fac800b6b50 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac86b9d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fac800d03e0 con 0x7fac6c0039b0 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac8539a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 0x7fac800b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.621+0000 7fac8539a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 0x7fac800b6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51440/0 (socket says 172.21.15.146:51440) 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 -- 172.21.15.146:0/2001690490 learned_addr learned my addr 172.21.15.146:0/2001690490 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0039b0 0x7fac80157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8639c640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac800b6b50 0x7fac801f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 -- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0039b0 msgr2=0x7fac80157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0039b0 0x7fac80157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 -- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac800b6b50 msgr2=0x7fac801f8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac800b6b50 0x7fac801f8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 -- 172.21.15.146:0/2001690490 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac800b6f60 con 0x7fac800cf1a0 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8639c640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac800b6b50 0x7fac801f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0039b0 0x7fac80157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac8539a640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 0x7fac800b6610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac70011660 tx=0x7fac70011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7000f9c0 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac7000fb60 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac800cd850 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac70015460 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.622+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac800ce030 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.623+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac80050740 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.624+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fac70015600 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.624+0000 7fac767fc640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac640744e0 0x7fac640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.624+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fac70095eb0 con 0x7fac800cf1a0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.624+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac640744e0 0x7fac640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.625+0000 7fac85b9b640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac640744e0 0x7fac640769f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac7c033000 tx=0x7fac7c038040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.629+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac800b6f60 con 0x7fac800cf1a0 2024-04-25T15:20:35.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.808+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fac8006d670 con 0x7fac800cf1a0 2024-04-25T15:20:35.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.808+0000 7fac767fc640 1 -- 172.21.15.146:0/2001690490 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fac70066070 con 0x7fac800cf1a0 2024-04-25T15:20:35.810 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:35.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac640744e0 msgr2=0x7fac640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac640744e0 0x7fac640769f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac7c033000 tx=0x7fac7c038040 comp rx=0 tx=0).stop 2024-04-25T15:20:35.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 msgr2=0x7fac800b6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 0x7fac800b6610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac70011660 tx=0x7fac70011b20 comp rx=0 tx=0).stop 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 shutdown_connections 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac640744e0 0x7fac640769f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac800b6b50 0x7fac801f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac800cf1a0 0x7fac800b6610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 --2- 172.21.15.146:0/2001690490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac6c0039b0 0x7fac80157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 >> 172.21.15.146:0/2001690490 conn(0x7fac800d2ba0 msgr2=0x7fac8006ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 shutdown_connections 2024-04-25T15:20:35.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.810+0000 7fac86b9d640 1 -- 172.21.15.146:0/2001690490 wait complete. 2024-04-25T15:20:35.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.933+0000 7f829b8ee640 1 Processor -- start 2024-04-25T15:20:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.933+0000 7f829b8ee640 1 -- start start 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.934+0000 7f829b8ee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f82940d2d90 0x7f829409bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f829b8ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f82940b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f829b8ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82940b1960 con 0x7f82940ccd40 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f829b8ee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f82940b3ce0 con 0x7f82940d2d90 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f829b8ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f82940b1c00 con 0x7f82940b2400 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f829968a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f82940ccd40 0x7f82940d2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42962/0 (socket says 172.21.15.146:42962) 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f8298e89640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f82940d2d90 0x7f829409bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f829968a640 1 -- 172.21.15.146:0/3040884062 learned_addr learned my addr 172.21.15.146:0/3040884062 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f8299e8b640 1 --2- 172.21.15.146:0/3040884062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f82940b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f8298e89640 1 -- 172.21.15.146:0/3040884062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 msgr2=0x7f82940b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f8298e89640 1 --2- 172.21.15.146:0/3040884062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f82940b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f8298e89640 1 -- 172.21.15.146:0/3040884062 >> v1:172.21.15.146:6789/0 conn(0x7f82940ccd40 legacy=0x7f82940d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.935+0000 7f8298e89640 1 -- 172.21.15.146:0/3040884062 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82940b1ee0 con 0x7f82940d2d90 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.936+0000 7f8298e89640 1 --2- 172.21.15.146:0/3040884062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82940d2d90 0x7f829409bba0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f82840097a0 tx=0x7f828402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4557185d9f2165 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.936+0000 7f828bfff640 1 -- 172.21.15.146:0/3040884062 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284037070 con 0x7f82940d2d90 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.936+0000 7f828bfff640 1 -- 172.21.15.146:0/3040884062 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8284032440 con 0x7f82940d2d90 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.936+0000 7f828bfff640 1 -- 172.21.15.146:0/3040884062 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284032810 con 0x7f82940d2d90 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.936+0000 7f829b8ee640 1 -- 172.21.15.146:0/3040884062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82940d2d90 msgr2=0x7f829409bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.936+0000 7f829b8ee640 1 --2- 172.21.15.146:0/3040884062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82940d2d90 0x7f829409bba0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f82840097a0 tx=0x7f828402a180 comp rx=0 tx=0).stop 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.937+0000 7f829b8ee640 1 -- 172.21.15.146:0/3040884062 shutdown_connections 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.937+0000 7f829b8ee640 1 --2- 172.21.15.146:0/3040884062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f82940b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.937+0000 7f829b8ee640 1 --2- 172.21.15.146:0/3040884062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82940d2d90 0x7f829409bba0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.937+0000 7f829b8ee640 1 -- 172.21.15.146:0/3040884062 >> 172.21.15.146:0/3040884062 conn(0x7f82940b7400 msgr2=0x7f82940b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.937+0000 7f829b8ee640 1 -- 172.21.15.146:0/3040884062 shutdown_connections 2024-04-25T15:20:35.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.937+0000 7f829b8ee640 1 -- 172.21.15.146:0/3040884062 wait complete. 2024-04-25T15:20:35.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.938+0000 7f829b8ee640 1 Processor -- start 2024-04-25T15:20:35.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.938+0000 7f829b8ee640 1 -- start start 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f8294155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82940ccd40 0x7f82940c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 0x7f8294162100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f82940b3ac0 con 0x7f8294161cf0 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f82940b3940 con 0x7f82940ccd40 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f829409b440 con 0x7f82940b2400 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 0x7f8294162100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 0x7f8294162100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51466/0 (socket says 172.21.15.146:51466) 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 -- 172.21.15.146:0/4110371230 learned_addr learned my addr 172.21.15.146:0/4110371230 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829968a640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f8294155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8298e89640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82940ccd40 0x7f82940c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 -- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 msgr2=0x7f8294155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f8294155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 -- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82940ccd40 msgr2=0x7f82940c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82940ccd40 0x7f82940c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 -- 172.21.15.146:0/4110371230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8294162640 con 0x7f8294161cf0 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8298e89640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82940ccd40 0x7f82940c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8299e8b640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 0x7f8294162100 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f828c007e90 tx=0x7f828c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c014420 con 0x7f8294161cf0 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f828c019070 con 0x7f8294161cf0 2024-04-25T15:20:35.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82940ca260 con 0x7f8294161cf0 2024-04-25T15:20:35.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.940+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c014800 con 0x7f8294161cf0 2024-04-25T15:20:35.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.941+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82940ca820 con 0x7f8294161cf0 2024-04-25T15:20:35.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.942+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829406a0e0 con 0x7f8294161cf0 2024-04-25T15:20:35.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.942+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f828c0149a0 con 0x7f8294161cf0 2024-04-25T15:20:35.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.943+0000 7f8289ffb640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82780744e0 0x7f82780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:35.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.943+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f828c095db0 con 0x7f8294161cf0 2024-04-25T15:20:35.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.943+0000 7f829968a640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82780744e0 0x7f82780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:35.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.943+0000 7f829968a640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82780744e0 0x7f82780769f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8290001000 tx=0x7f8290009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:35.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:35.947+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8294162640 con 0x7f8294161cf0 2024-04-25T15:20:36.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.123+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f82940c9ff0 con 0x7f8294161cf0 2024-04-25T15:20:36.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.123+0000 7f8289ffb640 1 -- 172.21.15.146:0/4110371230 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f828c060150 con 0x7f8294161cf0 2024-04-25T15:20:36.125 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.125+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82780744e0 msgr2=0x7f82780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82780744e0 0x7f82780769f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8290001000 tx=0x7f8290009040 comp rx=0 tx=0).stop 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 msgr2=0x7f8294162100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 0x7f8294162100 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f828c007e90 tx=0x7f828c00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 shutdown_connections 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f82780744e0 0x7f82780769f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8294161cf0 0x7f8294162100 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82940ccd40 0x7f82940c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 --2- 172.21.15.146:0/4110371230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82940b2400 0x7f8294155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 >> 172.21.15.146:0/4110371230 conn(0x7f82940b7400 msgr2=0x7f82940cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 shutdown_connections 2024-04-25T15:20:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.126+0000 7f829b8ee640 1 -- 172.21.15.146:0/4110371230 wait complete. 2024-04-25T15:20:36.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-25T15:20:36.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.247+0000 7f8bcf8c6640 1 Processor -- start 2024-04-25T15:20:36.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.248+0000 7f8bcf8c6640 1 -- start start 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcf8c6640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bc80d4f60 0x7f8bc80d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcf8c6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc80d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcf8c6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8bc8163d80 0x7f8bc8164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcf8c6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8bc806c550 con 0x7f8bc8163d80 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcf8c6640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8bc806c830 con 0x7f8bc80d4f60 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcf8c6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc80b9200 con 0x7f8bc80d0020 2024-04-25T15:20:36.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8bc8163d80 0x7f8bc8164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8bc8163d80 0x7f8bc8164190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51474/0 (socket says 172.21.15.146:51474) 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 -- 172.21.15.146:0/2173735890 learned_addr learned my addr 172.21.15.146:0/2173735890 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcce61640 1 --2- 172.21.15.146:0/2173735890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc80d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcd662640 1 --2- 172.21.15.146:0/2173735890 >> v2:172.21.15.177:3300/0 conn(0x7f8bc80d4f60 0x7f8bc80d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 -- 172.21.15.146:0/2173735890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 msgr2=0x7f8bc80d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 --2- 172.21.15.146:0/2173735890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc80d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 -- 172.21.15.146:0/2173735890 >> v2:172.21.15.177:3300/0 conn(0x7f8bc80d4f60 msgr2=0x7f8bc80d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 --2- 172.21.15.146:0/2173735890 >> v2:172.21.15.177:3300/0 conn(0x7f8bc80d4f60 0x7f8bc80d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.249+0000 7f8bcde63640 1 -- 172.21.15.146:0/2173735890 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc80d0970 con 0x7f8bc8163d80 2024-04-25T15:20:36.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.250+0000 7f8bcde63640 1 --2- 172.21.15.146:0/2173735890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc8164190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80d5930 tx=0x7f8bc002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe17492add5b967 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:36.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.250+0000 7f8bbffff640 1 -- 172.21.15.146:0/2173735890 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0036070 con 0x7f8bc8163d80 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.250+0000 7f8bbffff640 1 -- 172.21.15.146:0/2173735890 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc0031600 con 0x7f8bc8163d80 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.250+0000 7f8bbffff640 1 -- 172.21.15.146:0/2173735890 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc00319d0 con 0x7f8bc8163d80 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.251+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/2173735890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 msgr2=0x7f8bc8164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.251+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/2173735890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc8164190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80d5930 tx=0x7f8bc002a570 comp rx=0 tx=0).stop 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.251+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/2173735890 shutdown_connections 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.251+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/2173735890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc8164190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.251+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/2173735890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc80d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.251+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/2173735890 >> v2:172.21.15.177:3300/0 conn(0x7f8bc80d4f60 0x7f8bc80d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.252+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/2173735890 >> 172.21.15.146:0/2173735890 conn(0x7f8bc80d2450 msgr2=0x7f8bc80d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.252+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/2173735890 shutdown_connections 2024-04-25T15:20:36.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.252+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/2173735890 wait complete. 2024-04-25T15:20:36.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.254+0000 7f8bcf8c6640 1 Processor -- start 2024-04-25T15:20:36.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.255+0000 7f8bcf8c6640 1 -- start start 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcf8c6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcf8c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bc80d4f60 0x7f8bc815fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcf8c6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc81f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcf8c6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc80d15f0 con 0x7f8bc8163d80 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcf8c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc80d1070 con 0x7f8bc80d4f60 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcf8c6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc80b5150 con 0x7f8bc80d0020 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc81f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc81f64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51482/0 (socket says 172.21.15.146:51482) 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 -- 172.21.15.146:0/3441589945 learned_addr learned my addr 172.21.15.146:0/3441589945 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcd662640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcce61640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bc80d4f60 0x7f8bc815fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 -- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 msgr2=0x7f8bc815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc815cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 -- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bc80d4f60 msgr2=0x7f8bc815fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bc80d4f60 0x7f8bc815fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 -- 172.21.15.146:0/3441589945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc815d580 con 0x7f8bc8163d80 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcce61640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bc80d4f60 0x7f8bc815fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.257+0000 7f8bcde63640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc81f64e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8bc002ab60 tx=0x7f8bc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.258+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc003f020 con 0x7f8bc8163d80 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.258+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc0037040 con 0x7f8bc8163d80 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.258+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc815d860 con 0x7f8bc8163d80 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.258+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0036070 con 0x7f8bc8163d80 2024-04-25T15:20:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.258+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc815de20 con 0x7f8bc8163d80 2024-04-25T15:20:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.259+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc806a0e0 con 0x7f8bc8163d80 2024-04-25T15:20:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.260+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8bc0042020 con 0x7f8bc8163d80 2024-04-25T15:20:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.260+0000 7f8bbdffb640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.260+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f8bc00b59e0 con 0x7f8bc8163d80 2024-04-25T15:20:36.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.260+0000 7f8bcd662640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.261+0000 7f8bcd662640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80b4880 tx=0x7f8bb8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:36.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.265+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8bc0049030 con 0x7f8bc8163d80 2024-04-25T15:20:36.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.517+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7f8bc80d56b0 con 0x7f8bc8163d80 2024-04-25T15:20:36.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.672+0000 7f8bbdffb640 1 -- 172.21.15.146:0/3441589945 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v71) ==== 140+0+0 (secure 0 0 0) 0x7f8bc80d0970 con 0x7f8bc8163d80 2024-04-25T15:20:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 msgr2=0x7f8bac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80b4880 tx=0x7f8bb8008040 comp rx=0 tx=0).stop 2024-04-25T15:20:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 msgr2=0x7f8bc81f64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc81f64e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8bc002ab60 tx=0x7f8bc0004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 shutdown_connections 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8bac0744b0 0x7f8bac0769c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bc8163d80 0x7f8bc81f64e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bc80d4f60 0x7f8bc815fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 --2- 172.21.15.146:0/3441589945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bc80d0020 0x7f8bc815cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 >> 172.21.15.146:0/3441589945 conn(0x7f8bc80d2450 msgr2=0x7f8bc806cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 shutdown_connections 2024-04-25T15:20:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.674+0000 7f8bcf8c6640 1 -- 172.21.15.146:0/3441589945 wait complete. 2024-04-25T15:20:36.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:20:36.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.800+0000 7f4815938640 1 Processor -- start 2024-04-25T15:20:36.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.800+0000 7f4815938640 1 -- start start 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f4815938640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f48100b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f4815938640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f48100d2190 0x7f48100cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f4815938640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481008dd40 con 0x7f48100d1d80 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f4815938640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f481008e020 con 0x7f48100d2190 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f4815938640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f481008daf0 con 0x7f48100b6030 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f48100d2190 0x7f48100cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480e7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f48100d1d80 0x7f481006f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42968/0 (socket says 172.21.15.146:42968) 2024-04-25T15:20:36.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480e7fc640 1 -- 172.21.15.146:0/154090339 learned_addr learned my addr 172.21.15.146:0/154090339 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:36.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480f7fe640 1 -- 172.21.15.146:0/154090339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 msgr2=0x7f48100b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:36.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480f7fe640 1 --2- 172.21.15.146:0/154090339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f48100b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480f7fe640 1 -- 172.21.15.146:0/154090339 >> v1:172.21.15.146:6789/0 conn(0x7f48100d1d80 legacy=0x7f481006f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.802+0000 7f480f7fe640 1 -- 172.21.15.146:0/154090339 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48100d27b0 con 0x7f48100d2190 2024-04-25T15:20:36.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.803+0000 7f480f7fe640 1 --2- 172.21.15.146:0/154090339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cdf90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f480802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9069ab21d1203d81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:36.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.803+0000 7f480dffb640 1 -- 172.21.15.146:0/154090339 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808007740 con 0x7f48100d2190 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.803+0000 7f480dffb640 1 -- 172.21.15.146:0/154090339 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48080078e0 con 0x7f48100d2190 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.803+0000 7f480dffb640 1 -- 172.21.15.146:0/154090339 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808007cb0 con 0x7f48100d2190 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 -- 172.21.15.146:0/154090339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 msgr2=0x7f48100cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 --2- 172.21.15.146:0/154090339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cdf90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f480802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 -- 172.21.15.146:0/154090339 shutdown_connections 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 --2- 172.21.15.146:0/154090339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cdf90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 --2- 172.21.15.146:0/154090339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f48100b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 -- 172.21.15.146:0/154090339 >> 172.21.15.146:0/154090339 conn(0x7f48100b5430 msgr2=0x7f48100b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 -- 172.21.15.146:0/154090339 shutdown_connections 2024-04-25T15:20:36.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.804+0000 7f4815938640 1 -- 172.21.15.146:0/154090339 wait complete. 2024-04-25T15:20:36.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.806+0000 7f4815938640 1 Processor -- start 2024-04-25T15:20:36.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.806+0000 7f4815938640 1 -- start start 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.807+0000 7f4815938640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f4810163e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.807+0000 7f4815938640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48100d1d80 0x7f4810151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.807+0000 7f4815938640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.807+0000 7f4815938640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f48100d2c10 con 0x7f48100d2190 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.807+0000 7f4815938640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f48100d2a90 con 0x7f48100d1d80 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.807+0000 7f4815938640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f48100ce4d0 con 0x7f48100b6030 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51504/0 (socket says 172.21.15.146:51504) 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 -- 172.21.15.146:0/3583639945 learned_addr learned my addr 172.21.15.146:0/3583639945 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:36.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480effd640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f4810163e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480e7fc640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48100d1d80 0x7f4810151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 -- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 msgr2=0x7f4810163e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f4810163e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 -- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48100d1d80 msgr2=0x7f4810151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48100d1d80 0x7f4810151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 -- 172.21.15.146:0/3583639945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4810160bb0 con 0x7f48100d2190 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f480f7fe640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cc930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f4808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480803f070 con 0x7f48100d2190 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4808031c90 con 0x7f48100d2190 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4810160e90 con 0x7f48100d2190 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808043610 con 0x7f48100d2190 2024-04-25T15:20:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.808+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4810161450 con 0x7f48100d2190 2024-04-25T15:20:36.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.810+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f481006a0e0 con 0x7f48100d2190 2024-04-25T15:20:36.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.810+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f480803b410 con 0x7f48100d2190 2024-04-25T15:20:36.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.811+0000 7f47f7fff640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47fc0744b0 0x7f47fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:36.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.811+0000 7f480effd640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47fc0744b0 0x7f47fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:36.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.811+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f48080b6a60 con 0x7f48100d2190 2024-04-25T15:20:36.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.811+0000 7f480effd640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47fc0744b0 0x7f47fc0769c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f48100745d0 tx=0x7f4804009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:36.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.815+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4808080900 con 0x7f48100d2190 2024-04-25T15:20:36.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.993+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f48100cdd70 con 0x7f48100d2190 2024-04-25T15:20:37.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:36.999+0000 7f47f7fff640 1 -- 172.21.15.146:0/3583639945 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f4808084c00 con 0x7f48100d2190 2024-04-25T15:20:37.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47fc0744b0 msgr2=0x7f47fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47fc0744b0 0x7f47fc0769c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f48100745d0 tx=0x7f4804009060 comp rx=0 tx=0).stop 2024-04-25T15:20:37.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 msgr2=0x7f48100cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cc930 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f4808004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 shutdown_connections 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47fc0744b0 0x7f47fc0769c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48100d2190 0x7f48100cc930 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f48100d1d80 0x7f4810151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 --2- 172.21.15.146:0/3583639945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f48100b6030 0x7f4810163e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 >> 172.21.15.146:0/3583639945 conn(0x7f48100b5430 msgr2=0x7f48100684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.001+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 shutdown_connections 2024-04-25T15:20:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.002+0000 7f4815938640 1 -- 172.21.15.146:0/3583639945 wait complete. 2024-04-25T15:20:37.015 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:20:37.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:20:37.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:20:37.111 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:20:37.111 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:20:37.112 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:37.127 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:20:37.128 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:37.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:37.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.329+0000 7f77ab46e640 1 Processor -- start 2024-04-25T15:20:37.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.329+0000 7f77ab46e640 1 -- start start 2024-04-25T15:20:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.330+0000 7f77ab46e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77a40d4f60 0x7f77a40d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77ab46e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f77a406ccc0 0x7f77a406d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77ab46e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77a40b9200 con 0x7f77a40d0020 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77ab46e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f77a406c550 con 0x7f77a40d4f60 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77ab46e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f77a406c830 con 0x7f77a406ccc0 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77a40d4f60 0x7f77a40d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77a40d4f60 0x7f77a40d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51516/0 (socket says 172.21.15.146:51516) 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 -- 172.21.15.146:0/2741074374 learned_addr learned my addr 172.21.15.146:0/2741074374 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a9a0b640 1 --2- 172.21.15.146:0/2741074374 >> v2:172.21.15.177:3300/0 conn(0x7f77a406ccc0 0x7f77a406d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 -- 172.21.15.146:0/2741074374 >> v2:172.21.15.177:3300/0 conn(0x7f77a406ccc0 msgr2=0x7f77a406d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 --2- 172.21.15.146:0/2741074374 >> v2:172.21.15.177:3300/0 conn(0x7f77a406ccc0 0x7f77a406d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 -- 172.21.15.146:0/2741074374 >> v1:172.21.15.177:6789/0 conn(0x7f77a40d0020 legacy=0x7f77a40d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.331+0000 7f77a920a640 1 -- 172.21.15.146:0/2741074374 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a40d1070 con 0x7f77a40d4f60 2024-04-25T15:20:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.332+0000 7f77a920a640 1 --2- 172.21.15.146:0/2741074374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a40d5370 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f77a40d3e10 tx=0x7f779402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcf6a8b617cee4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.332+0000 7f779bfff640 1 -- 172.21.15.146:0/2741074374 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794037070 con 0x7f77a40d4f60 2024-04-25T15:20:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.332+0000 7f779bfff640 1 -- 172.21.15.146:0/2741074374 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7794032440 con 0x7f77a40d4f60 2024-04-25T15:20:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.332+0000 7f779bfff640 1 -- 172.21.15.146:0/2741074374 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794032810 con 0x7f77a40d4f60 2024-04-25T15:20:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 -- 172.21.15.146:0/2741074374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 msgr2=0x7f77a40d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2741074374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a40d5370 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f77a40d3e10 tx=0x7f779402a150 comp rx=0 tx=0).stop 2024-04-25T15:20:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 -- 172.21.15.146:0/2741074374 shutdown_connections 2024-04-25T15:20:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2741074374 >> v2:172.21.15.177:3300/0 conn(0x7f77a406ccc0 0x7f77a406d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2741074374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a40d5370 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 -- 172.21.15.146:0/2741074374 >> 172.21.15.146:0/2741074374 conn(0x7f77a40d2450 msgr2=0x7f77a40d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:37.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 -- 172.21.15.146:0/2741074374 shutdown_connections 2024-04-25T15:20:37.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.333+0000 7f77ab46e640 1 -- 172.21.15.146:0/2741074374 wait complete. 2024-04-25T15:20:37.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.335+0000 7f77ab46e640 1 Processor -- start 2024-04-25T15:20:37.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.335+0000 7f77ab46e640 1 -- start start 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77ab46e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77a406ccc0 0x7f77a40caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77ab46e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77a40d0020 0x7f77a40cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77ab46e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a4150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77ab46e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f77a40b5150 con 0x7f77a40d4f60 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77ab46e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77a40d15f0 con 0x7f77a40d0020 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77ab46e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f77a40b52d0 con 0x7f77a406ccc0 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a4150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a4150a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51526/0 (socket says 172.21.15.146:51526) 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 -- 172.21.15.146:0/2561387879 learned_addr learned my addr 172.21.15.146:0/2561387879 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a8a09640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77a40d0020 0x7f77a40cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a920a640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77a406ccc0 0x7f77a40caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 -- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77a406ccc0 msgr2=0x7f77a40caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77a406ccc0 0x7f77a40caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 -- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77a40d0020 msgr2=0x7f77a40cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77a40d0020 0x7f77a40cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a9a0b640 1 -- 172.21.15.146:0/2561387879 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a414ea80 con 0x7f77a40d4f60 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.336+0000 7f77a920a640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77a406ccc0 0x7f77a40caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.337+0000 7f77a9a0b640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a4150a50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f779c007d80 tx=0x7f779c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.337+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c014420 con 0x7f77a40d4f60 2024-04-25T15:20:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.337+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f779c019070 con 0x7f77a40d4f60 2024-04-25T15:20:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.337+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a40cb8b0 con 0x7f77a40d4f60 2024-04-25T15:20:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.337+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c014800 con 0x7f77a40d4f60 2024-04-25T15:20:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.337+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a40cbe70 con 0x7f77a40d4f60 2024-04-25T15:20:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.338+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a406a0e0 con 0x7f77a40d4f60 2024-04-25T15:20:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.338+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f779c0149a0 con 0x7f77a40d4f60 2024-04-25T15:20:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.339+0000 7f7799ffb640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77880744b0 0x7f77880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.339+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f779c095df0 con 0x7f77a40d4f60 2024-04-25T15:20:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.339+0000 7f77a920a640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77880744b0 0x7f77880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.339+0000 7f77a920a640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77880744b0 0x7f77880769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7794034000 tx=0x7f779402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.343+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f779c060190 con 0x7f77a40d4f60 2024-04-25T15:20:37.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.523+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77a40d4100 con 0x7f77a40d4f60 2024-04-25T15:20:37.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.524+0000 7f7799ffb640 1 -- 172.21.15.146:0/2561387879 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f779c05fa90 con 0x7f77a40d4f60 2024-04-25T15:20:37.526 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:37.526 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:37.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77880744b0 msgr2=0x7f77880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77880744b0 0x7f77880769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7794034000 tx=0x7f779402e000 comp rx=0 tx=0).stop 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 msgr2=0x7f77a4150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a4150a50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f779c007d80 tx=0x7f779c00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 shutdown_connections 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77880744b0 0x7f77880769c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a40d4f60 0x7f77a4150a50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77a40d0020 0x7f77a40cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 --2- 172.21.15.146:0/2561387879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f77a406ccc0 0x7f77a40caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 >> 172.21.15.146:0/2561387879 conn(0x7f77a40d2450 msgr2=0x7f77a406d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 shutdown_connections 2024-04-25T15:20:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.526+0000 7f77ab46e640 1 -- 172.21.15.146:0/2561387879 wait complete. 2024-04-25T15:20:37.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:37.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:37.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.649+0000 7f3faef4d640 1 Processor -- start 2024-04-25T15:20:37.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.649+0000 7f3faef4d640 1 -- start start 2024-04-25T15:20:37.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3faef4d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3fa80d0fb0 0x7f3fa80b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3faef4d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa80b5230 con 0x7f3fa80bb450 2024-04-25T15:20:37.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3faef4d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa806c8c0 con 0x7f3fa8163c00 2024-04-25T15:20:37.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3faef4d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3fa806cba0 con 0x7f3fa80d0fb0 2024-04-25T15:20:37.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3fad4ea640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3fa80d0fb0 0x7f3fa80b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3fad4ea640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3fa80d0fb0 0x7f3fa80b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51534/0 (socket says 172.21.15.146:51534) 2024-04-25T15:20:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.650+0000 7f3fad4ea640 1 -- 172.21.15.146:0/523766450 learned_addr learned my addr 172.21.15.146:0/523766450 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.651+0000 7f3fad4ea640 1 -- 172.21.15.146:0/523766450 >> v1:172.21.15.177:6789/0 conn(0x7f3fa8163c00 legacy=0x7f3fa806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.651+0000 7f3fad4ea640 1 -- 172.21.15.146:0/523766450 >> v1:172.21.15.146:6789/0 conn(0x7f3fa80bb450 legacy=0x7f3fa80d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.651+0000 7f3fad4ea640 1 -- 172.21.15.146:0/523766450 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa806ce80 con 0x7f3fa80d0fb0 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.651+0000 7f3fad4ea640 1 --2- 172.21.15.146:0/523766450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80d0fb0 0x7f3fa80b8fb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009e50 tx=0x7f3fa002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80b32b14e2ff3b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.651+0000 7f3fa77fe640 1 -- 172.21.15.146:0/523766450 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0036070 con 0x7f3fa80d0fb0 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3fa77fe640 1 -- 172.21.15.146:0/523766450 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fa0031600 con 0x7f3fa80d0fb0 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3fa77fe640 1 -- 172.21.15.146:0/523766450 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa00319d0 con 0x7f3fa80d0fb0 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 -- 172.21.15.146:0/523766450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80d0fb0 msgr2=0x7f3fa80b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 --2- 172.21.15.146:0/523766450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80d0fb0 0x7f3fa80b8fb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009e50 tx=0x7f3fa002a730 comp rx=0 tx=0).stop 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 -- 172.21.15.146:0/523766450 shutdown_connections 2024-04-25T15:20:37.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 --2- 172.21.15.146:0/523766450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80d0fb0 0x7f3fa80b8fb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 -- 172.21.15.146:0/523766450 >> 172.21.15.146:0/523766450 conn(0x7f3fa80b7970 msgr2=0x7f3fa80b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:37.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 -- 172.21.15.146:0/523766450 shutdown_connections 2024-04-25T15:20:37.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.652+0000 7f3faef4d640 1 -- 172.21.15.146:0/523766450 wait complete. 2024-04-25T15:20:37.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.654+0000 7f3faef4d640 1 Processor -- start 2024-04-25T15:20:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.654+0000 7f3faef4d640 1 -- start start 2024-04-25T15:20:37.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.655+0000 7f3faef4d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 0x7f3fa80ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.655+0000 7f3faef4d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3fa80d0fb0 0x7f3fa81f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.655+0000 7f3faef4d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fa8163c00 0x7f3fa81f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.655+0000 7f3faef4d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa80d13c0 con 0x7f3fa80bb450 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.655+0000 7f3faef4d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa806c550 con 0x7f3fa8163c00 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.655+0000 7f3faef4d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa80d1540 con 0x7f3fa80d0fb0 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 0x7f3fa80ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 0x7f3fa80ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51550/0 (socket says 172.21.15.146:51550) 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 -- 172.21.15.146:0/3497658701 learned_addr learned my addr 172.21.15.146:0/3497658701 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3fa7fff640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3fa80d0fb0 0x7f3fa81f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3fad4ea640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fa8163c00 0x7f3fa81f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 -- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3fa80d0fb0 msgr2=0x7f3fa81f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3fa80d0fb0 0x7f3fa81f5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 -- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fa8163c00 msgr2=0x7f3fa81f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fa8163c00 0x7f3fa81f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 -- 172.21.15.146:0/3497658701 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa814e960 con 0x7f3fa80bb450 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3facce9640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 0x7f3fa80ce380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c00c3f0 tx=0x7f3f9c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c0052d0 con 0x7f3fa80bb450 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f9c005490 con 0x7f3fa80bb450 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa80c95d0 con 0x7f3fa80bb450 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c007260 con 0x7f3fa80bb450 2024-04-25T15:20:37.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.656+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa815d190 con 0x7f3fa80bb450 2024-04-25T15:20:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.657+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa806a0e0 con 0x7f3fa80bb450 2024-04-25T15:20:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.658+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3f9c007400 con 0x7f3fa80bb450 2024-04-25T15:20:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.658+0000 7f3fa57fa640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f8c0744e0 0x7f3f8c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.659+0000 7f3fa7fff640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f8c0744e0 0x7f3f8c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.659+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f3f9c094a20 con 0x7f3fa80bb450 2024-04-25T15:20:37.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.659+0000 7f3fa7fff640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f8c0744e0 0x7f3f8c0769f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8159020 tx=0x7f3f94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.663+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f9c05e770 con 0x7f3fa80bb450 2024-04-25T15:20:37.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.844+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3fa80d3ed0 con 0x7f3fa80bb450 2024-04-25T15:20:37.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.845+0000 7f3fa57fa640 1 -- 172.21.15.146:0/3497658701 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3f9c062a70 con 0x7f3fa80bb450 2024-04-25T15:20:37.847 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.846+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f8c0744e0 msgr2=0x7f3f8c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.846+0000 7f3faef4d640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f8c0744e0 0x7f3f8c0769f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8159020 tx=0x7f3f94009040 comp rx=0 tx=0).stop 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 msgr2=0x7f3fa80ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 0x7f3fa80ce380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c00c3f0 tx=0x7f3f9c00c8b0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 shutdown_connections 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f8c0744e0 0x7f3f8c0769f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fa8163c00 0x7f3fa81f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3fa80d0fb0 0x7f3fa81f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 --2- 172.21.15.146:0/3497658701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa80bb450 0x7f3fa80ce380 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 >> 172.21.15.146:0/3497658701 conn(0x7f3fa80b7970 msgr2=0x7f3fa80b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 shutdown_connections 2024-04-25T15:20:37.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.847+0000 7f3faef4d640 1 -- 172.21.15.146:0/3497658701 wait complete. 2024-04-25T15:20:37.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:37.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.970+0000 7f0b9ef06640 1 Processor -- start 2024-04-25T15:20:37.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.970+0000 7f0b9ef06640 1 -- start start 2024-04-25T15:20:37.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.971+0000 7f0b9ef06640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b980b2cd0 0x7f0b980d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.971+0000 7f0b9ef06640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0b980d3300 0x7f0b98071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.971+0000 7f0b9ef06640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b98071b60 0x7f0b9808cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9ef06640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0b98071f70 con 0x7f0b980b2cd0 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9ef06640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0b980d2260 con 0x7f0b980d3300 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9ef06640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0b980d2540 con 0x7f0b98071b60 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b980b2cd0 0x7f0b980d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0b980b2cd0 0x7f0b980d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51552/0 (socket says 172.21.15.146:51552) 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 -- 172.21.15.146:0/2260713365 learned_addr learned my addr 172.21.15.146:0/2260713365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9d4a3640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b98071b60 0x7f0b9808cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b97fff640 1 --2- 172.21.15.146:0/2260713365 >> v2:172.21.15.177:3300/0 conn(0x7f0b980d3300 0x7f0b98071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 -- 172.21.15.146:0/2260713365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b98071b60 msgr2=0x7f0b9808cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b98071b60 0x7f0b9808cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 -- 172.21.15.146:0/2260713365 >> v2:172.21.15.177:3300/0 conn(0x7f0b980d3300 msgr2=0x7f0b98071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 --2- 172.21.15.146:0/2260713365 >> v2:172.21.15.177:3300/0 conn(0x7f0b980d3300 0x7f0b98071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9cca2640 1 -- 172.21.15.146:0/2260713365 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b980d2820 con 0x7f0b980b2cd0 2024-04-25T15:20:37.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b9d4a3640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b98071b60 0x7f0b9808cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:37.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.972+0000 7f0b97fff640 1 --2- 172.21.15.146:0/2260713365 >> v2:172.21.15.177:3300/0 conn(0x7f0b980d3300 0x7f0b98071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:37.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.973+0000 7f0b9cca2640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980b2cd0 0x7f0b980d1c80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b98075280 tx=0x7f0b8802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ecbd82a36a33b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.973+0000 7f0b977fe640 1 -- 172.21.15.146:0/2260713365 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88004030 con 0x7f0b980b2cd0 2024-04-25T15:20:37.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.973+0000 7f0b977fe640 1 -- 172.21.15.146:0/2260713365 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b880041d0 con 0x7f0b980b2cd0 2024-04-25T15:20:37.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.973+0000 7f0b977fe640 1 -- 172.21.15.146:0/2260713365 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88003610 con 0x7f0b980b2cd0 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2260713365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980b2cd0 msgr2=0x7f0b980d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980b2cd0 0x7f0b980d1c80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0b98075280 tx=0x7f0b8802a130 comp rx=0 tx=0).stop 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2260713365 shutdown_connections 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b98071b60 0x7f0b9808cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2260713365 >> v2:172.21.15.177:3300/0 conn(0x7f0b980d3300 0x7f0b98071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2260713365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980b2cd0 0x7f0b980d1c80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2260713365 >> 172.21.15.146:0/2260713365 conn(0x7f0b980b52e0 msgr2=0x7f0b980b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2260713365 shutdown_connections 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.974+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2260713365 wait complete. 2024-04-25T15:20:37.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.975+0000 7f0b9ef06640 1 Processor -- start 2024-04-25T15:20:37.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.975+0000 7f0b9ef06640 1 -- start start 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9ef06640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b98071b60 0x7f0b981f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9ef06640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b980b2cd0 0x7f0b980c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9ef06640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 0x7f0b9806e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9ef06640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0b980cfb70 con 0x7f0b980d3300 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9ef06640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b980d2de0 con 0x7f0b98071b60 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9ef06640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9808dbf0 con 0x7f0b980b2cd0 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 0x7f0b9806e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 0x7f0b9806e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51554/0 (socket says 172.21.15.146:51554) 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 -- 172.21.15.146:0/2758715020 learned_addr learned my addr 172.21.15.146:0/2758715020 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:37.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9cca2640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b98071b60 0x7f0b981f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b97fff640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b980b2cd0 0x7f0b980c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 -- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b980b2cd0 msgr2=0x7f0b980c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b980b2cd0 0x7f0b980c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 -- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b98071b60 msgr2=0x7f0b981f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b98071b60 0x7f0b981f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b9d4a3640 1 -- 172.21.15.146:0/2758715020 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b980c8730 con 0x7f0b980d3300 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.977+0000 7f0b97fff640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b980b2cd0 0x7f0b980c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.978+0000 7f0b9d4a3640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 0x7f0b9806e5b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000cca0 tx=0x7f0b9000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.978+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9000f680 con 0x7f0b980d3300 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.978+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9000f840 con 0x7f0b980d3300 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.978+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b980c8a10 con 0x7f0b980d3300 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.978+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9000fc30 con 0x7f0b980d3300 2024-04-25T15:20:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.978+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b980c8fd0 con 0x7f0b980d3300 2024-04-25T15:20:37.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.979+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b9806a0e0 con 0x7f0b980d3300 2024-04-25T15:20:37.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.979+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0b900213f0 con 0x7f0b980d3300 2024-04-25T15:20:37.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.980+0000 7f0b957fa640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b840744b0 0x7f0b840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:37.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.980+0000 7f0b9cca2640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b840744b0 0x7f0b840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:37.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.980+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f0b90096270 con 0x7f0b980d3300 2024-04-25T15:20:37.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.980+0000 7f0b9cca2640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b840744b0 0x7f0b840769c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0b88033000 tx=0x7f0b8802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:37.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:37.985+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b90014030 con 0x7f0b980d3300 2024-04-25T15:20:38.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.161+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0b98068a50 con 0x7f0b980d3300 2024-04-25T15:20:38.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.162+0000 7f0b957fa640 1 -- 172.21.15.146:0/2758715020 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f0b900606a0 con 0x7f0b980d3300 2024-04-25T15:20:38.164 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b840744b0 msgr2=0x7f0b840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b840744b0 0x7f0b840769c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0b88033000 tx=0x7f0b8802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 msgr2=0x7f0b9806e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 0x7f0b9806e5b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b9000cca0 tx=0x7f0b9000a650 comp rx=0 tx=0).stop 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 shutdown_connections 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0b840744b0 0x7f0b840769c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b980d3300 0x7f0b9806e5b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b980b2cd0 0x7f0b980c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 --2- 172.21.15.146:0/2758715020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b98071b60 0x7f0b981f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 >> 172.21.15.146:0/2758715020 conn(0x7f0b980b52e0 msgr2=0x7f0b980b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 shutdown_connections 2024-04-25T15:20:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.164+0000 7f0b9ef06640 1 -- 172.21.15.146:0/2758715020 wait complete. 2024-04-25T15:20:38.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:38.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.285+0000 7f606d0b6640 1 Processor -- start 2024-04-25T15:20:38.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.285+0000 7f606d0b6640 1 -- start start 2024-04-25T15:20:38.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606d0b6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680b2cd0 0x7f60680d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606d0b6640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f60680d3300 0x7f6068071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606d0b6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6068071f70 con 0x7f6068071b60 2024-04-25T15:20:38.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606d0b6640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f60680d2260 con 0x7f60680d3300 2024-04-25T15:20:38.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606d0b6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f60680d2540 con 0x7f60680b2cd0 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606659c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f60680d3300 0x7f6068071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f6066d9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680b2cd0 0x7f60680d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606659c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f60680d3300 0x7f6068071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48388/0 (socket says 172.21.15.146:48388) 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606759e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f6068071b60 0x7f606808cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:47260/0 (socket says 172.21.15.146:47260) 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606659c640 1 -- 172.21.15.146:0/1397764448 learned_addr learned my addr 172.21.15.146:0/1397764448 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606659c640 1 -- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680b2cd0 msgr2=0x7f60680d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606659c640 1 --2- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680b2cd0 0x7f60680d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.287+0000 7f606659c640 1 -- 172.21.15.146:0/1397764448 >> v1:172.21.15.177:6789/0 conn(0x7f6068071b60 legacy=0x7f606808cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.288+0000 7f606659c640 1 -- 172.21.15.146:0/1397764448 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60680d2820 con 0x7f60680d3300 2024-04-25T15:20:38.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.288+0000 7f6066d9d640 1 --2- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680b2cd0 0x7f60680d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:38.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.288+0000 7f606659c640 1 --2- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680d3300 0x7f6068071620 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f605c009870 tx=0x7f605c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d30c043c29934 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.288+0000 7f6065d9b640 1 -- 172.21.15.146:0/1397764448 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c037070 con 0x7f60680d3300 2024-04-25T15:20:38.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.288+0000 7f6065d9b640 1 -- 172.21.15.146:0/1397764448 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605c032440 con 0x7f60680d3300 2024-04-25T15:20:38.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.288+0000 7f6065d9b640 1 -- 172.21.15.146:0/1397764448 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c032810 con 0x7f60680d3300 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 -- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680d3300 msgr2=0x7f6068071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 --2- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680d3300 0x7f6068071620 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f605c009870 tx=0x7f605c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 -- 172.21.15.146:0/1397764448 shutdown_connections 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 --2- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680d3300 0x7f6068071620 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 --2- 172.21.15.146:0/1397764448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680b2cd0 0x7f60680d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 -- 172.21.15.146:0/1397764448 >> 172.21.15.146:0/1397764448 conn(0x7f60680b52e0 msgr2=0x7f60680b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 -- 172.21.15.146:0/1397764448 shutdown_connections 2024-04-25T15:20:38.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.289+0000 7f606d0b6640 1 -- 172.21.15.146:0/1397764448 wait complete. 2024-04-25T15:20:38.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.291+0000 7f606d0b6640 1 Processor -- start 2024-04-25T15:20:38.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.291+0000 7f606d0b6640 1 -- start start 2024-04-25T15:20:38.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 0x7f606815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680b2cd0 0x7f60681f98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680d3300 0x7f60681f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f60680cfb70 con 0x7f6068071b60 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60680d2de0 con 0x7f60680b2cd0 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f606808dbf0 con 0x7f60680d3300 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 0x7f606815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 0x7f606815ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51562/0 (socket says 172.21.15.146:51562) 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 -- 172.21.15.146:0/3805650755 learned_addr learned my addr 172.21.15.146:0/3805650755 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606759e640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680d3300 0x7f60681f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606659c640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680b2cd0 0x7f60681f98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 -- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680d3300 msgr2=0x7f60681f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680d3300 0x7f60681f9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 -- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680b2cd0 msgr2=0x7f60681f98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680b2cd0 0x7f60681f98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 -- 172.21.15.146:0/3805650755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606815c2a0 con 0x7f6068071b60 2024-04-25T15:20:38.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f6066d9d640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 0x7f606815ace0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f605800efc0 tx=0x7f605800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058018020 con 0x7f6068071b60 2024-04-25T15:20:38.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6058014680 con 0x7f6068071b60 2024-04-25T15:20:38.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60680b1440 con 0x7f6068071b60 2024-04-25T15:20:38.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.293+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058014a50 con 0x7f6068071b60 2024-04-25T15:20:38.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.294+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60680b1a00 con 0x7f6068071b60 2024-04-25T15:20:38.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.294+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606806a0e0 con 0x7f6068071b60 2024-04-25T15:20:38.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.295+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f605801b020 con 0x7f6068071b60 2024-04-25T15:20:38.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.296+0000 7f604f7fe640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60540744b0 0x7f60540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.296+0000 7f606659c640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60540744b0 0x7f60540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.296+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f6058096330 con 0x7f6068071b60 2024-04-25T15:20:38.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.296+0000 7f606659c640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60540744b0 0x7f60540769c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f605c034000 tx=0x7f605c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.300+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6058022020 con 0x7f6068071b60 2024-04-25T15:20:38.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.474+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6068068790 con 0x7f6068071b60 2024-04-25T15:20:38.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.475+0000 7f604f7fe640 1 -- 172.21.15.146:0/3805650755 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f6058010040 con 0x7f6068071b60 2024-04-25T15:20:38.476 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:38.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60540744b0 msgr2=0x7f60540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60540744b0 0x7f60540769c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f605c034000 tx=0x7f605c02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 msgr2=0x7f606815ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 0x7f606815ace0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f605800efc0 tx=0x7f605800c490 comp rx=0 tx=0).stop 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 shutdown_connections 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60540744b0 0x7f60540769c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60680d3300 0x7f60681f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60680b2cd0 0x7f60681f98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.476+0000 7f606d0b6640 1 --2- 172.21.15.146:0/3805650755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6068071b60 0x7f606815ace0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.477+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 >> 172.21.15.146:0/3805650755 conn(0x7f60680b52e0 msgr2=0x7f60680723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.477+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 shutdown_connections 2024-04-25T15:20:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.477+0000 7f606d0b6640 1 -- 172.21.15.146:0/3805650755 wait complete. 2024-04-25T15:20:38.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:38.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.600+0000 7fc512700640 1 Processor -- start 2024-04-25T15:20:38.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.600+0000 7fc512700640 1 -- start start 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.601+0000 7fc512700640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc50c15d480 0x7fc50c073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.601+0000 7fc512700640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc50c0b6460 0x7fc50c074020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.601+0000 7fc512700640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc50c15d940 con 0x7fc50c0ae660 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.601+0000 7fc512700640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc50c06fe30 con 0x7fc50c0b6460 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.601+0000 7fc512700640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc50c0702c0 con 0x7fc50c15d480 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc50c0b6460 0x7fc50c074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc50c0b6460 0x7fc50c074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51566/0 (socket says 172.21.15.146:51566) 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 -- 172.21.15.146:0/4159355053 learned_addr learned my addr 172.21.15.146:0/4159355053 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc50bfff640 1 --2- 172.21.15.146:0/4159355053 >> v2:172.21.15.177:3300/0 conn(0x7fc50c15d480 0x7fc50c073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 -- 172.21.15.146:0/4159355053 >> v2:172.21.15.177:3300/0 conn(0x7fc50c15d480 msgr2=0x7fc50c073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 --2- 172.21.15.146:0/4159355053 >> v2:172.21.15.177:3300/0 conn(0x7fc50c15d480 0x7fc50c073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 -- 172.21.15.146:0/4159355053 >> v1:172.21.15.146:6789/0 conn(0x7fc50c0ae660 legacy=0x7fc50c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.602+0000 7fc510c9d640 1 -- 172.21.15.146:0/4159355053 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c06f500 con 0x7fc50c0b6460 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc510c9d640 1 --2- 172.21.15.146:0/4159355053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c074020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc50c072300 tx=0x7fc50402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3a06f4ee55c27a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc50affd640 1 -- 172.21.15.146:0/4159355053 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504035610 con 0x7fc50c0b6460 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc50affd640 1 -- 172.21.15.146:0/4159355053 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5040357b0 con 0x7fc50c0b6460 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc50affd640 1 -- 172.21.15.146:0/4159355053 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504035b80 con 0x7fc50c0b6460 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc512700640 1 -- 172.21.15.146:0/4159355053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 msgr2=0x7fc50c074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc512700640 1 --2- 172.21.15.146:0/4159355053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c074020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc50c072300 tx=0x7fc50402c8b0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc512700640 1 -- 172.21.15.146:0/4159355053 shutdown_connections 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc512700640 1 --2- 172.21.15.146:0/4159355053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c074020 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc512700640 1 --2- 172.21.15.146:0/4159355053 >> v2:172.21.15.177:3300/0 conn(0x7fc50c15d480 0x7fc50c073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.603+0000 7fc512700640 1 -- 172.21.15.146:0/4159355053 >> 172.21.15.146:0/4159355053 conn(0x7fc50c0d1c10 msgr2=0x7fc50c0733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:38.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.604+0000 7fc512700640 1 -- 172.21.15.146:0/4159355053 shutdown_connections 2024-04-25T15:20:38.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.604+0000 7fc512700640 1 -- 172.21.15.146:0/4159355053 wait complete. 2024-04-25T15:20:38.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.605+0000 7fc512700640 1 Processor -- start 2024-04-25T15:20:38.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.605+0000 7fc512700640 1 -- start start 2024-04-25T15:20:38.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc50c0ae660 0x7fc50c157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c1442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc50c158f50 0x7fc50c1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c0b5620 con 0x7fc50c0b6460 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c06fac0 con 0x7fc50c158f50 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc50c0d0f10 con 0x7fc50c0ae660 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c1442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c1442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51570/0 (socket says 172.21.15.146:51570) 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 -- 172.21.15.146:0/2735434683 learned_addr learned my addr 172.21.15.146:0/2735434683 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc510c9d640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc50c158f50 0x7fc50c1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50bfff640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc50c0ae660 0x7fc50c157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 -- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc50c0ae660 msgr2=0x7fc50c157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc50c0ae660 0x7fc50c157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 -- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc50c158f50 msgr2=0x7fc50c1596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc50c158f50 0x7fc50c1596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 -- 172.21.15.146:0/2735434683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c1578c0 con 0x7fc50c0b6460 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc50b7fe640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c1442b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc50000c3f0 tx=0x7fc50000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000052d0 con 0x7fc50c0b6460 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500005490 con 0x7fc50c0b6460 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.607+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c157ba0 con 0x7fc50c0b6460 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.608+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500007260 con 0x7fc50c0b6460 2024-04-25T15:20:38.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.608+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c158160 con 0x7fc50c0b6460 2024-04-25T15:20:38.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.609+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc50c06a0e0 con 0x7fc50c0b6460 2024-04-25T15:20:38.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.609+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc500007400 con 0x7fc50c0b6460 2024-04-25T15:20:38.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.609+0000 7fc508ff9640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4f80744e0 0x7fc4f80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.610+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fc500094930 con 0x7fc50c0b6460 2024-04-25T15:20:38.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.610+0000 7fc50bfff640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4f80744e0 0x7fc4f80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.610+0000 7fc50bfff640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4f80744e0 0x7fc4f80769f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009970 tx=0x7fc4fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.614+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc50005edf0 con 0x7fc50c0b6460 2024-04-25T15:20:38.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.792+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc50c070c50 con 0x7fc50c0b6460 2024-04-25T15:20:38.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.792+0000 7fc508ff9640 1 -- 172.21.15.146:0/2735434683 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fc50005e6f0 con 0x7fc50c0b6460 2024-04-25T15:20:38.794 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:38.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4f80744e0 msgr2=0x7fc4f80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4f80744e0 0x7fc4f80769f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009970 tx=0x7fc4fc008040 comp rx=0 tx=0).stop 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 msgr2=0x7fc50c1442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c1442b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc50000c3f0 tx=0x7fc50000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 shutdown_connections 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4f80744e0 0x7fc4f80769f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc50c158f50 0x7fc50c1596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc50c0b6460 0x7fc50c1442b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 --2- 172.21.15.146:0/2735434683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc50c0ae660 0x7fc50c157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.794+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 >> 172.21.15.146:0/2735434683 conn(0x7fc50c0d1c10 msgr2=0x7fc50c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.795+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 shutdown_connections 2024-04-25T15:20:38.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.795+0000 7fc512700640 1 -- 172.21.15.146:0/2735434683 wait complete. 2024-04-25T15:20:38.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:38.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.916+0000 7f16a2508640 1 Processor -- start 2024-04-25T15:20:38.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.916+0000 7f16a2508640 1 -- start start 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.917+0000 7f16a2508640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 0x7f169c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.917+0000 7f16a2508640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c0cc7e0 con 0x7f169c0d0420 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.917+0000 7f16a2508640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c06e0a0 con 0x7f169c161540 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.917+0000 7f16a2508640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f169c06e380 con 0x7f169c161130 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f169c0d0420 0x7f169c160a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42996/0 (socket says 172.21.15.146:42996) 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169bfff640 1 -- 172.21.15.146:0/2998787155 learned_addr learned my addr 172.21.15.146:0/2998787155 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169b7fe640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 0x7f169c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666432130 0 0) 0x7f169c0cc7e0 con 0x7f169c0d0420 2024-04-25T15:20:38.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1684003520 con 0x7f169c0d0420 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1484841240 0 0) 0x7f1684003520 con 0x7f169c0d0420 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 msgr2=0x7f169c0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 0x7f169c0cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 >> v1:172.21.15.177:6789/0 conn(0x7f169c161540 legacy=0x7f169c0cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c06e660 con 0x7f169c0d0420 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f168c001670 con 0x7f169c0d0420 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 >> v1:172.21.15.146:6789/0 conn(0x7f169c0d0420 legacy=0x7f169c160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16840039e0 0x7f1684003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 0x7f1684004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1684004d40 0x7f1684005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f169c06e380 con 0x7f1684004d40 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f168400bb50 con 0x7f16840039e0 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f168400df10 con 0x7f169c161130 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1684004d40 0x7f1684005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169bfff640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16840039e0 0x7f1684003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.918+0000 7f169b7fe640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 0x7f1684004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.919+0000 7f16a0aa5640 1 -- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 msgr2=0x7f1684004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.919+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f169c161130 0x7f1684004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.919+0000 7f16a0aa5640 1 -- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16840039e0 msgr2=0x7f1684003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.919+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16840039e0 0x7f1684003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.919+0000 7f16a0aa5640 1 -- 172.21.15.146:0/2998787155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1684003520 con 0x7f1684004d40 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.919+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1684004d40 0x7f1684005150 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f169c072660 tx=0x7f1684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd91417de083dc2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694001b30 con 0x7f1684004d40 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1694031620 con 0x7f1684004d40 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f169affd640 1 -- 172.21.15.146:0/2998787155 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694031a10 con 0x7f1684004d40 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a2508640 1 -- 172.21.15.146:0/2998787155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1684004d40 msgr2=0x7f1684005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a2508640 1 --2- 172.21.15.146:0/2998787155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1684004d40 0x7f1684005150 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f169c072660 tx=0x7f1684005830 comp rx=0 tx=0).stop 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a0aa5640 1 -- 172.21.15.146:0/2998787155 reap_dead start 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a2508640 1 -- 172.21.15.146:0/2998787155 shutdown_connections 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a2508640 1 -- 172.21.15.146:0/2998787155 >> 172.21.15.146:0/2998787155 conn(0x7f169c09b2b0 msgr2=0x7f169c0b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:38.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a2508640 1 -- 172.21.15.146:0/2998787155 shutdown_connections 2024-04-25T15:20:38.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.920+0000 7f16a2508640 1 -- 172.21.15.146:0/2998787155 wait complete. 2024-04-25T15:20:38.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.922+0000 7f16a2508640 1 Processor -- start 2024-04-25T15:20:38.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.922+0000 7f16a2508640 1 -- start start 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.923+0000 7f16a2508640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1684004d40 0x7f169c15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.923+0000 7f16a2508640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f169c0c69c0 0x7f169c15f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.923+0000 7f16a2508640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 0x7f169c15fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.923+0000 7f16a2508640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0d09b0 con 0x7f169c0c99d0 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.923+0000 7f16a2508640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0d0830 con 0x7f169c0c69c0 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.923+0000 7f16a2508640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f169c15d1d0 con 0x7f1684004d40 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 0x7f169c15fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 0x7f169c15fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51592/0 (socket says 172.21.15.146:51592) 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 -- 172.21.15.146:0/236731039 learned_addr learned my addr 172.21.15.146:0/236731039 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f169b7fe640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f169c0c69c0 0x7f169c15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f169bfff640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1684004d40 0x7f169c15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 -- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1684004d40 msgr2=0x7f169c15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1684004d40 0x7f169c15dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 -- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f169c0c69c0 msgr2=0x7f169c15f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f169c0c69c0 0x7f169c15f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 -- 172.21.15.146:0/236731039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c0c7130 con 0x7f169c0c99d0 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a0aa5640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 0x7f169c15fd90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f169402a9c0 tx=0x7f16940015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694006d60 con 0x7f169c0c99d0 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16940317d0 con 0x7f169c0c99d0 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c0c7410 con 0x7f169c0c99d0 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694031b80 con 0x7f169c0c99d0 2024-04-25T15:20:38.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.924+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c0c79d0 con 0x7f169c0c99d0 2024-04-25T15:20:38.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.925+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c0cc070 con 0x7f169c0c99d0 2024-04-25T15:20:38.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.926+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1694044020 con 0x7f169c0c99d0 2024-04-25T15:20:38.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.926+0000 7f1698ff9640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16880744b0 0x7f16880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:38.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.927+0000 7f169bfff640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16880744b0 0x7f16880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:38.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.927+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f16940b5d50 con 0x7f169c0c99d0 2024-04-25T15:20:38.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.927+0000 7f169bfff640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16880744b0 0x7f16880769c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f168c000c00 tx=0x7f168c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:38.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:38.931+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f169c0cc070 con 0x7f169c0c99d0 2024-04-25T15:20:39.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.119+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f169c06a0e0 con 0x7f169c0c99d0 2024-04-25T15:20:39.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.120+0000 7f1698ff9640 1 -- 172.21.15.146:0/236731039 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f16940b9030 con 0x7f169c0c99d0 2024-04-25T15:20:39.121 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16880744b0 msgr2=0x7f16880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16880744b0 0x7f16880769c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f168c000c00 tx=0x7f168c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 msgr2=0x7f169c15fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 0x7f169c15fd90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f169402a9c0 tx=0x7f16940015b0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 shutdown_connections 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16880744b0 0x7f16880769c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0c99d0 0x7f169c15fd90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f169c0c69c0 0x7f169c15f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 --2- 172.21.15.146:0/236731039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1684004d40 0x7f169c15dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 >> 172.21.15.146:0/236731039 conn(0x7f169c09b2b0 msgr2=0x7f169c06cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 shutdown_connections 2024-04-25T15:20:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.122+0000 7f16a2508640 1 -- 172.21.15.146:0/236731039 wait complete. 2024-04-25T15:20:39.135 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:20:39.135 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:39.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:39.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.296+0000 7fbb5cf6c640 1 Processor -- start 2024-04-25T15:20:39.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.296+0000 7fbb5cf6c640 1 -- start start 2024-04-25T15:20:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5cf6c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb580b7350 0x7fbb580715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5cf6c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbb58160000 0x7fbb5806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5cf6c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb58070190 0x7fbb580ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5cf6c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbb580705a0 con 0x7fbb58070190 2024-04-25T15:20:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5cf6c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbb58071bc0 con 0x7fbb58160000 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5cf6c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbb580d1b00 con 0x7fbb580b7350 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb58070190 0x7fbb580ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb58070190 0x7fbb580ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51598/0 (socket says 172.21.15.146:51598) 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 -- 172.21.15.146:0/1516290988 learned_addr learned my addr 172.21.15.146:0/1516290988 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb4ffff640 1 --2- 172.21.15.146:0/1516290988 >> v2:172.21.15.177:3300/0 conn(0x7fbb58160000 0x7fbb5806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb5659c640 1 --2- 172.21.15.146:0/1516290988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb580b7350 0x7fbb580715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 -- 172.21.15.146:0/1516290988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb580b7350 msgr2=0x7fbb580715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 --2- 172.21.15.146:0/1516290988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb580b7350 0x7fbb580715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 -- 172.21.15.146:0/1516290988 >> v2:172.21.15.177:3300/0 conn(0x7fbb58160000 msgr2=0x7fbb5806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 --2- 172.21.15.146:0/1516290988 >> v2:172.21.15.177:3300/0 conn(0x7fbb58160000 0x7fbb5806fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.297+0000 7fbb56d9d640 1 -- 172.21.15.146:0/1516290988 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb580d1de0 con 0x7fbb58070190 2024-04-25T15:20:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.298+0000 7fbb56d9d640 1 --2- 172.21.15.146:0/1516290988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb580ce4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbb580b4a70 tx=0x7fbb5002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dcf28e37ff7d58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.298+0000 7fbb55d9b640 1 -- 172.21.15.146:0/1516290988 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50036070 con 0x7fbb58070190 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.298+0000 7fbb55d9b640 1 -- 172.21.15.146:0/1516290988 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb50031600 con 0x7fbb58070190 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.298+0000 7fbb55d9b640 1 -- 172.21.15.146:0/1516290988 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500319d0 con 0x7fbb58070190 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1516290988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 msgr2=0x7fbb580ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1516290988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb580ce4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbb580b4a70 tx=0x7fbb5002a610 comp rx=0 tx=0).stop 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1516290988 shutdown_connections 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1516290988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb580ce4a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1516290988 >> v2:172.21.15.177:3300/0 conn(0x7fbb58160000 0x7fbb5806fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1516290988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb580b7350 0x7fbb580715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1516290988 >> 172.21.15.146:0/1516290988 conn(0x7fbb5809b2b0 msgr2=0x7fbb580b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1516290988 shutdown_connections 2024-04-25T15:20:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.299+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1516290988 wait complete. 2024-04-25T15:20:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.301+0000 7fbb5cf6c640 1 Processor -- start 2024-04-25T15:20:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.301+0000 7fbb5cf6c640 1 -- start start 2024-04-25T15:20:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5cf6c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb5814b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5cf6c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb580b7350 0x7fbb5809c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5cf6c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb58160000 0x7fbb5809cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5cf6c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbb58070980 con 0x7fbb58070190 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5cf6c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbb580d23a0 con 0x7fbb580b7350 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5cf6c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbb58160410 con 0x7fbb58160000 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb5814b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb5814b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51614/0 (socket says 172.21.15.146:51614) 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.302+0000 7fbb5659c640 1 -- 172.21.15.146:0/1273726695 learned_addr learned my addr 172.21.15.146:0/1273726695 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb4ffff640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb580b7350 0x7fbb5809c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb56d9d640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb58160000 0x7fbb5809cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5659c640 1 -- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb58160000 msgr2=0x7fbb5809cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5659c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb58160000 0x7fbb5809cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5659c640 1 -- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb580b7350 msgr2=0x7fbb5809c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5659c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb580b7350 0x7fbb5809c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5659c640 1 -- 172.21.15.146:0/1273726695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5814bcc0 con 0x7fbb58070190 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb4ffff640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb580b7350 0x7fbb5809c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5659c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb5814b470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800d5f0 tx=0x7fbb4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004240 con 0x7fbb58070190 2024-04-25T15:20:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb480043e0 con 0x7fbb58070190 2024-04-25T15:20:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5814b9b0 con 0x7fbb58070190 2024-04-25T15:20:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48003610 con 0x7fbb58070190 2024-04-25T15:20:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.303+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5809d830 con 0x7fbb58070190 2024-04-25T15:20:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.304+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5806a0e0 con 0x7fbb58070190 2024-04-25T15:20:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.305+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbb4801b020 con 0x7fbb58070190 2024-04-25T15:20:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.305+0000 7fbb4effd640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb380744b0 0x7fbb380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.305+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fbb48095340 con 0x7fbb58070190 2024-04-25T15:20:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.305+0000 7fbb4ffff640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb380744b0 0x7fbb380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.306+0000 7fbb4ffff640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb380744b0 0x7fbb380769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb580b4550 tx=0x7fbb40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.310+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb4805f770 con 0x7fbb58070190 2024-04-25T15:20:39.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.485+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb580711b0 con 0x7fbb58070190 2024-04-25T15:20:39.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.486+0000 7fbb4effd640 1 -- 172.21.15.146:0/1273726695 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7fbb4805f070 con 0x7fbb58070190 2024-04-25T15:20:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:39.488 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:39.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb380744b0 msgr2=0x7fbb380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb380744b0 0x7fbb380769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb580b4550 tx=0x7fbb40008040 comp rx=0 tx=0).stop 2024-04-25T15:20:39.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 msgr2=0x7fbb5814b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb5814b470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800d5f0 tx=0x7fbb4800dab0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 shutdown_connections 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb380744b0 0x7fbb380769c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb58160000 0x7fbb5809cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb580b7350 0x7fbb5809c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 --2- 172.21.15.146:0/1273726695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb58070190 0x7fbb5814b470 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 >> 172.21.15.146:0/1273726695 conn(0x7fbb5809b2b0 msgr2=0x7fbb580cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 shutdown_connections 2024-04-25T15:20:39.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.488+0000 7fbb5cf6c640 1 -- 172.21.15.146:0/1273726695 wait complete. 2024-04-25T15:20:39.502 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:39.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:39.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.611+0000 7ff5529ee640 1 Processor -- start 2024-04-25T15:20:39.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.611+0000 7ff5529ee640 1 -- start start 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff5529ee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff54c0b4670 0x7ff54c0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff5529ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54c06c5f0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff5529ee640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54c1647b0 con 0x7ff54c0d1140 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff5529ee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff54c0b4a80 con 0x7ff54c0b4670 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff550f8b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff54c0b4670 0x7ff54c0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff550f8b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff54c0b4670 0x7ff54c0b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51618/0 (socket says 172.21.15.146:51618) 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff550f8b640 1 -- 172.21.15.146:0/2817388009 learned_addr learned my addr 172.21.15.146:0/2817388009 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff550f8b640 1 -- 172.21.15.146:0/2817388009 >> v1:172.21.15.177:6789/0 conn(0x7ff54c0d1140 legacy=0x7ff54c06d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff550f8b640 1 -- 172.21.15.146:0/2817388009 >> v1:172.21.15.146:6789/0 conn(0x7ff54c0b4c30 legacy=0x7ff54c0d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.613+0000 7ff550f8b640 1 -- 172.21.15.146:0/2817388009 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c06c890 con 0x7ff54c0b4670 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.614+0000 7ff550f8b640 1 --2- 172.21.15.146:0/2817388009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4670 0x7ff54c0b9110 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff544009e80 tx=0x7ff54402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471ff815eb3247ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.614+0000 7ff54affd640 1 -- 172.21.15.146:0/2817388009 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544036070 con 0x7ff54c0b4670 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.614+0000 7ff54affd640 1 -- 172.21.15.146:0/2817388009 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff544031600 con 0x7ff54c0b4670 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.614+0000 7ff54affd640 1 -- 172.21.15.146:0/2817388009 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5440319d0 con 0x7ff54c0b4670 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 -- 172.21.15.146:0/2817388009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4670 msgr2=0x7ff54c0b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 --2- 172.21.15.146:0/2817388009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4670 0x7ff54c0b9110 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff544009e80 tx=0x7ff54402a710 comp rx=0 tx=0).stop 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 -- 172.21.15.146:0/2817388009 shutdown_connections 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 --2- 172.21.15.146:0/2817388009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4670 0x7ff54c0b9110 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 -- 172.21.15.146:0/2817388009 >> 172.21.15.146:0/2817388009 conn(0x7ff54c0b7970 msgr2=0x7ff54c0b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 -- 172.21.15.146:0/2817388009 shutdown_connections 2024-04-25T15:20:39.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.615+0000 7ff5529ee640 1 -- 172.21.15.146:0/2817388009 wait complete. 2024-04-25T15:20:39.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.616+0000 7ff5529ee640 1 Processor -- start 2024-04-25T15:20:39.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.617+0000 7ff5529ee640 1 -- start start 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff5529ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff54c0b4670 0x7ff54c1f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff5529ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 0x7ff54c15e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff5529ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff54c0cc980 0x7ff54c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff5529ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c0d16b0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff5529ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c06cc00 con 0x7ff54c0cc980 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff5529ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff54c1600d0 con 0x7ff54c0b4670 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 0x7ff54c15e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 0x7ff54c15e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51630/0 (socket says 172.21.15.146:51630) 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 -- 172.21.15.146:0/4246025361 learned_addr learned my addr 172.21.15.146:0/4246025361 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:39.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54bfff640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff54c0b4670 0x7ff54c1f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff550f8b640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff54c0cc980 0x7ff54c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 -- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff54c0b4670 msgr2=0x7ff54c1f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff54c0b4670 0x7ff54c1f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 -- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff54c0cc980 msgr2=0x7ff54c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff54c0cc980 0x7ff54c0ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 -- 172.21.15.146:0/4246025361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c0ac550 con 0x7ff54c0b4c30 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff550f8b640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff54c0cc980 0x7ff54c0ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.618+0000 7ff54b7fe640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 0x7ff54c15e6a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff54000b3e0 tx=0x7ff54000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.619+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540015020 con 0x7ff54c0b4c30 2024-04-25T15:20:39.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.619+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff540011830 con 0x7ff54c0b4c30 2024-04-25T15:20:39.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.619+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540011c20 con 0x7ff54c0b4c30 2024-04-25T15:20:39.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.619+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54c0ac830 con 0x7ff54c0b4c30 2024-04-25T15:20:39.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.619+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54c0acdf0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.620+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54c06a0e0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.620+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff540018020 con 0x7ff54c0b4c30 2024-04-25T15:20:39.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.621+0000 7ff548ff9640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff5380744e0 0x7ff5380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.621+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7ff5400954b0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.621+0000 7ff54bfff640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff5380744e0 0x7ff5380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.621+0000 7ff54bfff640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff5380744e0 0x7ff5380769f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff54c08e090 tx=0x7ff53c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.625+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff54001f020 con 0x7ff54c0b4c30 2024-04-25T15:20:39.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.801+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff54c0d3fa0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.801+0000 7ff548ff9640 1 -- 172.21.15.146:0/4246025361 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7ff54005f8e0 con 0x7ff54c0b4c30 2024-04-25T15:20:39.803 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff5380744e0 msgr2=0x7ff5380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff5380744e0 0x7ff5380769f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff54c08e090 tx=0x7ff53c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 msgr2=0x7ff54c15e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 0x7ff54c15e6a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff54000b3e0 tx=0x7ff54000b8a0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 shutdown_connections 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff5380744e0 0x7ff5380769f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff54c0cc980 0x7ff54c0ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0b4c30 0x7ff54c15e6a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 --2- 172.21.15.146:0/4246025361 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff54c0b4670 0x7ff54c1f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 >> 172.21.15.146:0/4246025361 conn(0x7ff54c0b7970 msgr2=0x7ff54c15eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:39.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 shutdown_connections 2024-04-25T15:20:39.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.803+0000 7ff5529ee640 1 -- 172.21.15.146:0/4246025361 wait complete. 2024-04-25T15:20:39.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.926+0000 7f5f04ad8640 1 Processor -- start 2024-04-25T15:20:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.926+0000 7f5f04ad8640 1 -- start start 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.927+0000 7f5f04ad8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5f000ccd40 0x7f5f0009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.927+0000 7f5f04ad8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000d2680 0x7f5f000d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.927+0000 7f5f04ad8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f000b2400 0x7f5f000b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.927+0000 7f5f04ad8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5f000d2fd0 con 0x7f5f000b2400 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.927+0000 7f5f04ad8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5f000b1960 con 0x7f5f000ccd40 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.927+0000 7f5f04ad8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5f000b3ce0 con 0x7f5f000d2680 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f000b2400 0x7f5f000b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f000b2400 0x7f5f000b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51642/0 (socket says 172.21.15.146:51642) 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 -- 172.21.15.146:0/3466876024 learned_addr learned my addr 172.21.15.146:0/3466876024 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 -- 172.21.15.146:0/3466876024 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000d2680 msgr2=0x7f5f000d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efdd9b640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000d2680 0x7f5f000d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efe59c640 1 --2- 172.21.15.146:0/3466876024 >> v2:172.21.15.177:3300/0 conn(0x7f5f000ccd40 0x7f5f0009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000d2680 0x7f5f000d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 -- 172.21.15.146:0/3466876024 >> v2:172.21.15.177:3300/0 conn(0x7f5f000ccd40 msgr2=0x7f5f0009bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 --2- 172.21.15.146:0/3466876024 >> v2:172.21.15.177:3300/0 conn(0x7f5f000ccd40 0x7f5f0009bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efed9d640 1 -- 172.21.15.146:0/3466876024 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f000b1c00 con 0x7f5f000b2400 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efe59c640 1 --2- 172.21.15.146:0/3466876024 >> v2:172.21.15.177:3300/0 conn(0x7f5f000ccd40 0x7f5f0009bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.928+0000 7f5efdd9b640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000d2680 0x7f5f000d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.929+0000 7f5efed9d640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000b2400 0x7f5f000b2810 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5f000d2650 tx=0x7f5ef002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188fa034187f45e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.929+0000 7f5efd59a640 1 -- 172.21.15.146:0/3466876024 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0036070 con 0x7f5f000b2400 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.929+0000 7f5efd59a640 1 -- 172.21.15.146:0/3466876024 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef0031600 con 0x7f5f000b2400 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.929+0000 7f5efd59a640 1 -- 172.21.15.146:0/3466876024 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00319d0 con 0x7f5f000b2400 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.929+0000 7f5f04ad8640 1 -- 172.21.15.146:0/3466876024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000b2400 msgr2=0x7f5f000b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.929+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000b2400 0x7f5f000b2810 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5f000d2650 tx=0x7f5ef002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 -- 172.21.15.146:0/3466876024 shutdown_connections 2024-04-25T15:20:39.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000b2400 0x7f5f000b2810 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/3466876024 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000d2680 0x7f5f000d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/3466876024 >> v2:172.21.15.177:3300/0 conn(0x7f5f000ccd40 0x7f5f0009bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 -- 172.21.15.146:0/3466876024 >> 172.21.15.146:0/3466876024 conn(0x7f5f000b7400 msgr2=0x7f5f000b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:39.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 -- 172.21.15.146:0/3466876024 shutdown_connections 2024-04-25T15:20:39.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.930+0000 7f5f04ad8640 1 -- 172.21.15.146:0/3466876024 wait complete. 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.931+0000 7f5f04ad8640 1 Processor -- start 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.932+0000 7f5f04ad8640 1 -- start start 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5f04ad8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000b2400 0x7f5f00162ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5f04ad8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 0x7f5f000cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5f04ad8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f000d2680 0x7f5f0015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5f04ad8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5f000b3940 con 0x7f5f000ccd40 2024-04-25T15:20:39.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5f04ad8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5f000b3fc0 con 0x7f5f000d2680 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5f04ad8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5f000b3ac0 con 0x7f5f000b2400 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 0x7f5f000cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 0x7f5f000cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51650/0 (socket says 172.21.15.146:51650) 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 -- 172.21.15.146:0/1643805780 learned_addr learned my addr 172.21.15.146:0/1643805780 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efed9d640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f000d2680 0x7f5f0015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efe59c640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000b2400 0x7f5f00162ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 -- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000b2400 msgr2=0x7f5f00162ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000b2400 0x7f5f00162ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 -- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f000d2680 msgr2=0x7f5f0015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f000d2680 0x7f5f0015ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 -- 172.21.15.146:0/1643805780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f001633e0 con 0x7f5f000ccd40 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.933+0000 7f5efdd9b640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 0x7f5f000cb910 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800c7f0 tx=0x7f5ee800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.934+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8015070 con 0x7f5f000ccd40 2024-04-25T15:20:39.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.934+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee8007820 con 0x7f5f000ccd40 2024-04-25T15:20:39.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.934+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f000cc670 con 0x7f5f000ccd40 2024-04-25T15:20:39.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.934+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8007bd0 con 0x7f5f000ccd40 2024-04-25T15:20:39.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.934+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f00160a90 con 0x7f5f000ccd40 2024-04-25T15:20:39.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.935+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0006a0e0 con 0x7f5f000ccd40 2024-04-25T15:20:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.935+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5ee8018070 con 0x7f5f000ccd40 2024-04-25T15:20:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.936+0000 7f5ef6ffd640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ee00744b0 0x7f5ee00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.936+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5ee80957e0 con 0x7f5f000ccd40 2024-04-25T15:20:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.936+0000 7f5efe59c640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ee00744b0 0x7f5ee00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.936+0000 7f5efe59c640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ee00744b0 0x7f5ee00769c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f000b8040 tx=0x7f5ef8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:39.941+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ee805fb00 con 0x7f5f000ccd40 2024-04-25T15:20:40.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.116+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5f000cc400 con 0x7f5f000ccd40 2024-04-25T15:20:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.116+0000 7f5ef6ffd640 1 -- 172.21.15.146:0/1643805780 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f5ee805f400 con 0x7f5f000ccd40 2024-04-25T15:20:40.118 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:40.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ee00744b0 msgr2=0x7f5ee00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ee00744b0 0x7f5ee00769c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f000b8040 tx=0x7f5ef8008040 comp rx=0 tx=0).stop 2024-04-25T15:20:40.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 msgr2=0x7f5f000cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 0x7f5f000cb910 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800c7f0 tx=0x7f5ee800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 shutdown_connections 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ee00744b0 0x7f5ee00769c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f000d2680 0x7f5f0015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f000ccd40 0x7f5f000cb910 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 --2- 172.21.15.146:0/1643805780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f000b2400 0x7f5f00162ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 >> 172.21.15.146:0/1643805780 conn(0x7f5f000b7400 msgr2=0x7f5f0015cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 shutdown_connections 2024-04-25T15:20:40.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.118+0000 7f5f04ad8640 1 -- 172.21.15.146:0/1643805780 wait complete. 2024-04-25T15:20:40.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:40.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.240+0000 7fbfacb4d640 1 Processor -- start 2024-04-25T15:20:40.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.240+0000 7fbfacb4d640 1 -- start start 2024-04-25T15:20:40.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfacb4d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa80d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfacb4d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbfa8071d30 0x7fbfa80cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfacb4d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa80d2260 con 0x7fbfa80d3300 2024-04-25T15:20:40.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfacb4d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbfa80d2540 con 0x7fbfa8071d30 2024-04-25T15:20:40.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfacb4d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa80d2820 con 0x7fbfa80b2cd0 2024-04-25T15:20:40.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfa6d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbfa8071d30 0x7fbfa80cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfa6d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbfa8071d30 0x7fbfa80cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51660/0 (socket says 172.21.15.146:51660) 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.241+0000 7fbfa6d9d640 1 -- 172.21.15.146:0/2593699862 learned_addr learned my addr 172.21.15.146:0/2593699862 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa659c640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa80d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa6d9d640 1 -- 172.21.15.146:0/2593699862 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 msgr2=0x7fbfa80d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa6d9d640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa80d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa6d9d640 1 -- 172.21.15.146:0/2593699862 >> v1:172.21.15.146:6789/0 conn(0x7fbfa80d3300 legacy=0x7fbfa8071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa6d9d640 1 -- 172.21.15.146:0/2593699862 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa80d2b00 con 0x7fbfa8071d30 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa659c640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa80d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa6d9d640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa8071d30 0x7fbfa80cde40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009e50 tx=0x7fbfa002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57fa390c79c48df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.242+0000 7fbfa559a640 1 -- 172.21.15.146:0/2593699862 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0036070 con 0x7fbfa8071d30 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfa559a640 1 -- 172.21.15.146:0/2593699862 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa0031600 con 0x7fbfa8071d30 2024-04-25T15:20:40.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfa559a640 1 -- 172.21.15.146:0/2593699862 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa00319d0 con 0x7fbfa8071d30 2024-04-25T15:20:40.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 -- 172.21.15.146:0/2593699862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa8071d30 msgr2=0x7fbfa80cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa8071d30 0x7fbfa80cde40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009e50 tx=0x7fbfa002a680 comp rx=0 tx=0).stop 2024-04-25T15:20:40.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 -- 172.21.15.146:0/2593699862 shutdown_connections 2024-04-25T15:20:40.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa8071d30 0x7fbfa80cde40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/2593699862 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa80d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 -- 172.21.15.146:0/2593699862 >> 172.21.15.146:0/2593699862 conn(0x7fbfa80b52e0 msgr2=0x7fbfa80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:40.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 -- 172.21.15.146:0/2593699862 shutdown_connections 2024-04-25T15:20:40.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.243+0000 7fbfacb4d640 1 -- 172.21.15.146:0/2593699862 wait complete. 2024-04-25T15:20:40.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.245+0000 7fbfacb4d640 1 Processor -- start 2024-04-25T15:20:40.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.245+0000 7fbfacb4d640 1 -- start start 2024-04-25T15:20:40.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa81583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 0x7fbfa80b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa80b1410 0x7fbfa80b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa808dbf0 con 0x7fbfa80d3300 2024-04-25T15:20:40.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80cfb70 con 0x7fbfa80b1410 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa808dd70 con 0x7fbfa80b2cd0 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 0x7fbfa80b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 0x7fbfa80b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51670/0 (socket says 172.21.15.146:51670) 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 -- 172.21.15.146:0/1223299861 learned_addr learned my addr 172.21.15.146:0/1223299861 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa6d9d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa80b1410 0x7fbfa80b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 -- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 msgr2=0x7fbfa81583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa81583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 -- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa80b1410 msgr2=0x7fbfa80b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa80b1410 0x7fbfa80b1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 -- 172.21.15.146:0/1223299861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa80b1d60 con 0x7fbfa80d3300 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfa5d9b640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 0x7fbfa80b7df0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00c3c0 tx=0x7fbf9c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0052d0 con 0x7fbfa80d3300 2024-04-25T15:20:40.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf9c005470 con 0x7fbfa80d3300 2024-04-25T15:20:40.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa815b6a0 con 0x7fbfa80d3300 2024-04-25T15:20:40.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c007240 con 0x7fbfa80d3300 2024-04-25T15:20:40.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.247+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa8160830 con 0x7fbfa80d3300 2024-04-25T15:20:40.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.249+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa815bbc0 con 0x7fbfa80d3300 2024-04-25T15:20:40.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.249+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbf9c0073e0 con 0x7fbfa80d3300 2024-04-25T15:20:40.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.250+0000 7fbf96ffd640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf880744e0 0x7fbf880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.250+0000 7fbfa659c640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf880744e0 0x7fbf880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.250+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fbf9c094b00 con 0x7fbfa80d3300 2024-04-25T15:20:40.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.250+0000 7fbfa659c640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf880744e0 0x7fbf880769f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80b9490 tx=0x7fbf98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.254+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfa815bbc0 con 0x7fbfa80d3300 2024-04-25T15:20:40.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.429+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbfa806a0e0 con 0x7fbfa80d3300 2024-04-25T15:20:40.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.430+0000 7fbf96ffd640 1 -- 172.21.15.146:0/1223299861 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fbfa80b1d60 con 0x7fbfa80d3300 2024-04-25T15:20:40.432 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf880744e0 msgr2=0x7fbf880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf880744e0 0x7fbf880769f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80b9490 tx=0x7fbf98008040 comp rx=0 tx=0).stop 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 msgr2=0x7fbfa80b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 0x7fbfa80b7df0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00c3c0 tx=0x7fbf9c00c7c0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 shutdown_connections 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf880744e0 0x7fbf880769f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa80b1410 0x7fbfa80b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa80d3300 0x7fbfa80b7df0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 --2- 172.21.15.146:0/1223299861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa80b2cd0 0x7fbfa81583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 >> 172.21.15.146:0/1223299861 conn(0x7fbfa80b52e0 msgr2=0x7fbfa808c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:40.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 shutdown_connections 2024-04-25T15:20:40.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.432+0000 7fbfacb4d640 1 -- 172.21.15.146:0/1223299861 wait complete. 2024-04-25T15:20:40.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:40.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.554+0000 7fca787b7640 1 Processor -- start 2024-04-25T15:20:40.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.554+0000 7fca787b7640 1 -- start start 2024-04-25T15:20:40.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca787b7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fca700b0fa0 0x7fca7015e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca787b7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca700b3070 0x7fca700b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca787b7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700d1e30 0x7fca700d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca787b7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fca700b13b0 con 0x7fca700b3070 2024-04-25T15:20:40.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca787b7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fca700d2780 con 0x7fca700b0fa0 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca787b7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca700aee10 con 0x7fca700d1e30 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca75d52640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca700b3070 0x7fca700b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.555+0000 7fca75d52640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca700b3070 0x7fca700b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55864/0 (socket says 172.21.15.146:55864) 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 -- 172.21.15.146:0/3591740504 learned_addr learned my addr 172.21.15.146:0/3591740504 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca76553640 1 --2- 172.21.15.146:0/3591740504 >> v2:172.21.15.177:3300/0 conn(0x7fca700b0fa0 0x7fca7015e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca76d54640 1 --2- 172.21.15.146:0/3591740504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700d1e30 0x7fca700d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 -- 172.21.15.146:0/3591740504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700d1e30 msgr2=0x7fca700d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 --2- 172.21.15.146:0/3591740504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700d1e30 0x7fca700d2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 -- 172.21.15.146:0/3591740504 >> v2:172.21.15.177:3300/0 conn(0x7fca700b0fa0 msgr2=0x7fca7015e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 --2- 172.21.15.146:0/3591740504 >> v2:172.21.15.177:3300/0 conn(0x7fca700b0fa0 0x7fca7015e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 -- 172.21.15.146:0/3591740504 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca700af0b0 con 0x7fca700b3070 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca76553640 1 --2- 172.21.15.146:0/3591740504 >> v2:172.21.15.177:3300/0 conn(0x7fca700b0fa0 0x7fca7015e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.556+0000 7fca75d52640 1 --2- 172.21.15.146:0/3591740504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca700b3480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fca700cf980 tx=0x7fca6002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b8f767701283cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca75551640 1 -- 172.21.15.146:0/3591740504 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60037070 con 0x7fca700b3070 2024-04-25T15:20:40.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca75551640 1 -- 172.21.15.146:0/3591740504 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca60032440 con 0x7fca700b3070 2024-04-25T15:20:40.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca75551640 1 -- 172.21.15.146:0/3591740504 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60032810 con 0x7fca700b3070 2024-04-25T15:20:40.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 -- 172.21.15.146:0/3591740504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 msgr2=0x7fca700b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 --2- 172.21.15.146:0/3591740504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca700b3480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fca700cf980 tx=0x7fca6002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 -- 172.21.15.146:0/3591740504 shutdown_connections 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 --2- 172.21.15.146:0/3591740504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700d1e30 0x7fca700d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 --2- 172.21.15.146:0/3591740504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca700b3480 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 --2- 172.21.15.146:0/3591740504 >> v2:172.21.15.177:3300/0 conn(0x7fca700b0fa0 0x7fca7015e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.557+0000 7fca787b7640 1 -- 172.21.15.146:0/3591740504 >> 172.21.15.146:0/3591740504 conn(0x7fca7009b2b0 msgr2=0x7fca700b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.558+0000 7fca787b7640 1 -- 172.21.15.146:0/3591740504 shutdown_connections 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.558+0000 7fca787b7640 1 -- 172.21.15.146:0/3591740504 wait complete. 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.559+0000 7fca787b7640 1 Processor -- start 2024-04-25T15:20:40.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.559+0000 7fca787b7640 1 -- start start 2024-04-25T15:20:40.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.560+0000 7fca787b7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700b0fa0 0x7fca7015a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.560+0000 7fca787b7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca7015a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca787b7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca700d1e30 0x7fca7006c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca787b7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fca700af950 con 0x7fca700b3070 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca787b7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca700af670 con 0x7fca700d1e30 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca787b7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca700cf650 con 0x7fca700b0fa0 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca7015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca7015a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55876/0 (socket says 172.21.15.146:55876) 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 -- 172.21.15.146:0/3122897535 learned_addr learned my addr 172.21.15.146:0/3122897535 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca76553640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700b0fa0 0x7fca7015a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca76d54640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca700d1e30 0x7fca7006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 -- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700b0fa0 msgr2=0x7fca7015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700b0fa0 0x7fca7015a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 -- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca700d1e30 msgr2=0x7fca7006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca700d1e30 0x7fca7006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 -- 172.21.15.146:0/3122897535 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca70158a80 con 0x7fca700b3070 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca75d52640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca7015a970 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fca6002a220 tx=0x7fca600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60037040 con 0x7fca700b3070 2024-04-25T15:20:40.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca60042070 con 0x7fca700b3070 2024-04-25T15:20:40.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca70159390 con 0x7fca700b3070 2024-04-25T15:20:40.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60030bb0 con 0x7fca700b3070 2024-04-25T15:20:40.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.561+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca70159950 con 0x7fca700b3070 2024-04-25T15:20:40.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.562+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca7006a0e0 con 0x7fca700b3070 2024-04-25T15:20:40.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.563+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fca60047460 con 0x7fca700b3070 2024-04-25T15:20:40.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.563+0000 7fca66ffd640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca540744e0 0x7fca540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.564+0000 7fca76553640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca540744e0 0x7fca540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.564+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fca600b6290 con 0x7fca700b3070 2024-04-25T15:20:40.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.564+0000 7fca76553640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca540744e0 0x7fca540769f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca70161bb0 tx=0x7fca6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.568+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca6007ffe0 con 0x7fca700b3070 2024-04-25T15:20:40.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.745+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fca700719f0 con 0x7fca700b3070 2024-04-25T15:20:40.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.745+0000 7fca66ffd640 1 -- 172.21.15.146:0/3122897535 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fca600842e0 con 0x7fca700b3070 2024-04-25T15:20:40.747 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca540744e0 msgr2=0x7fca540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca540744e0 0x7fca540769f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fca70161bb0 tx=0x7fca6c008040 comp rx=0 tx=0).stop 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 msgr2=0x7fca7015a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca7015a970 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fca6002a220 tx=0x7fca600046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 shutdown_connections 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca540744e0 0x7fca540769f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca700d1e30 0x7fca7006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca700b3070 0x7fca7015a970 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 --2- 172.21.15.146:0/3122897535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca700b0fa0 0x7fca7015a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 >> 172.21.15.146:0/3122897535 conn(0x7fca7009b2b0 msgr2=0x7fca70070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:40.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 shutdown_connections 2024-04-25T15:20:40.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.747+0000 7fca787b7640 1 -- 172.21.15.146:0/3122897535 wait complete. 2024-04-25T15:20:40.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:40.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.867+0000 7f32529ba640 1 Processor -- start 2024-04-25T15:20:40.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.867+0000 7f32529ba640 1 -- start start 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.868+0000 7f32529ba640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f324c0b5950 0x7f324c0d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.868+0000 7f32529ba640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c15ffa0 0x7f324c0b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.868+0000 7f32529ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f324c0d2230 con 0x7f324c0b6610 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.868+0000 7f32529ba640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f324c0b2f50 con 0x7f324c15ffa0 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.868+0000 7f32529ba640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f324c0b2ac0 con 0x7f324c0b5950 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f324bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f324c0b6610 0x7f324c0b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34612/0 (socket says 172.21.15.146:34612) 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f324bfff640 1 -- 172.21.15.146:0/223482196 learned_addr learned my addr 172.21.15.146:0/223482196 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f3250f57640 1 --2- 172.21.15.146:0/223482196 >> v2:172.21.15.146:3300/0 conn(0x7f324c15ffa0 0x7f324c0b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f324b7fe640 1 --2- 172.21.15.146:0/223482196 >> v2:172.21.15.177:3300/0 conn(0x7f324c0b5950 0x7f324c0d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f3250f57640 1 -- 172.21.15.146:0/223482196 >> v2:172.21.15.177:3300/0 conn(0x7f324c0b5950 msgr2=0x7f324c0d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:40.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f3250f57640 1 --2- 172.21.15.146:0/223482196 >> v2:172.21.15.177:3300/0 conn(0x7f324c0b5950 0x7f324c0d2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f3250f57640 1 -- 172.21.15.146:0/223482196 >> v1:172.21.15.146:6789/0 conn(0x7f324c0b6610 legacy=0x7f324c0b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.869+0000 7f3250f57640 1 -- 172.21.15.146:0/223482196 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c0cfa80 con 0x7f324c15ffa0 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f3250f57640 1 --2- 172.21.15.146:0/223482196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c15ffa0 0x7f324c0b7cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f324c0c7080 tx=0x7f324402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48bfb0939ec5d65b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f324affd640 1 -- 172.21.15.146:0/223482196 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244007740 con 0x7f324c15ffa0 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f324affd640 1 -- 172.21.15.146:0/223482196 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32440078e0 con 0x7f324c15ffa0 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f324affd640 1 -- 172.21.15.146:0/223482196 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244007cb0 con 0x7f324c15ffa0 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f32529ba640 1 -- 172.21.15.146:0/223482196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c15ffa0 msgr2=0x7f324c0b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f32529ba640 1 --2- 172.21.15.146:0/223482196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c15ffa0 0x7f324c0b7cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f324c0c7080 tx=0x7f324402a680 comp rx=0 tx=0).stop 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f32529ba640 1 -- 172.21.15.146:0/223482196 shutdown_connections 2024-04-25T15:20:40.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f32529ba640 1 --2- 172.21.15.146:0/223482196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c15ffa0 0x7f324c0b7cc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f32529ba640 1 --2- 172.21.15.146:0/223482196 >> v2:172.21.15.177:3300/0 conn(0x7f324c0b5950 0x7f324c0d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.870+0000 7f32529ba640 1 -- 172.21.15.146:0/223482196 >> 172.21.15.146:0/223482196 conn(0x7f324c0d0240 msgr2=0x7f324c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:40.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.871+0000 7f32529ba640 1 -- 172.21.15.146:0/223482196 shutdown_connections 2024-04-25T15:20:40.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.871+0000 7f32529ba640 1 -- 172.21.15.146:0/223482196 wait complete. 2024-04-25T15:20:40.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.872+0000 7f32529ba640 1 Processor -- start 2024-04-25T15:20:40.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.872+0000 7f32529ba640 1 -- start start 2024-04-25T15:20:40.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 0x7f324c1f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c15ffa0 0x7f324c15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c158fe0 0x7f324c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f324c0d1da0 con 0x7f324c0b6610 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1603b0 con 0x7f324c158fe0 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f324c0d1f20 con 0x7f324c15ffa0 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 0x7f324c1f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 0x7f324c1f5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55886/0 (socket says 172.21.15.146:55886) 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 -- 172.21.15.146:0/337971717 learned_addr learned my addr 172.21.15.146:0/337971717 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 -- 172.21.15.146:0/337971717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c15ffa0 msgr2=0x7f324c15af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c15ffa0 0x7f324c15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 -- 172.21.15.146:0/337971717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c158fe0 msgr2=0x7f324c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c158fe0 0x7f324c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 -- 172.21.15.146:0/337971717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c159930 con 0x7f324c0b6610 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f324bfff640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 0x7f324c1f5d10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c3c0 tx=0x7f323c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c017070 con 0x7f324c0b6610 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f323c005ce0 con 0x7f324c0b6610 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324c1484f0 con 0x7f324c0b6610 2024-04-25T15:20:40.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c0072f0 con 0x7f324c0b6610 2024-04-25T15:20:40.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.874+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324c15bdc0 con 0x7f324c0b6610 2024-04-25T15:20:40.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.875+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f324c06a0e0 con 0x7f324c0b6610 2024-04-25T15:20:40.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.876+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f323c01a070 con 0x7f324c0b6610 2024-04-25T15:20:40.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.877+0000 7f3248ff9640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32380744e0 0x7f32380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:40.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.877+0000 7f324b7fe640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32380744e0 0x7f32380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:40.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.877+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f323c094840 con 0x7f324c0b6610 2024-04-25T15:20:40.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.877+0000 7f324b7fe640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32380744e0 0x7f32380769f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3240006fd0 tx=0x7f3240008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:40.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:40.881+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f324c159930 con 0x7f324c0b6610 2024-04-25T15:20:41.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.069+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f324c0cf2d0 con 0x7f324c0b6610 2024-04-25T15:20:41.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.069+0000 7f3248ff9640 1 -- 172.21.15.146:0/337971717 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f323c05ed00 con 0x7f324c0b6610 2024-04-25T15:20:41.071 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:41.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32380744e0 msgr2=0x7f32380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32380744e0 0x7f32380769f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3240006fd0 tx=0x7f3240008040 comp rx=0 tx=0).stop 2024-04-25T15:20:41.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 msgr2=0x7f324c1f5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 0x7f324c1f5d10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c3c0 tx=0x7f323c00c880 comp rx=0 tx=0).stop 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 shutdown_connections 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32380744e0 0x7f32380769f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c158fe0 0x7f324c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c15ffa0 0x7f324c15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 --2- 172.21.15.146:0/337971717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0b6610 0x7f324c1f5d10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 >> 172.21.15.146:0/337971717 conn(0x7f324c0d0240 msgr2=0x7f324c0b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 shutdown_connections 2024-04-25T15:20:41.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.071+0000 7f32529ba640 1 -- 172.21.15.146:0/337971717 wait complete. 2024-04-25T15:20:41.085 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:20:41.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:41.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T15:20:41.109 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:20:41.109 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:20:41.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:41.109 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:20:41.165 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:20:41.165 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:41.184 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:20:41.185 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:41.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:41.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.350+0000 7f0bd1338640 1 Processor -- start 2024-04-25T15:20:41.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.350+0000 7f0bd1338640 1 -- start start 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bd1338640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 0x7f0bcc0d2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bd1338640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0bcc0d2aa0 0x7f0bcc06def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bd1338640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bcc0b3fc0 con 0x7f0bcc15cfd0 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bd1338640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0bcc0cefe0 con 0x7f0bcc0d2aa0 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bd1338640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0bcc0d2eb0 con 0x7f0bcc0b3bb0 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bcaffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0bcc15cfd0 0x7f0bcc09bba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34622/0 (socket says 172.21.15.146:34622) 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bcaffd640 1 -- 172.21.15.146:0/2292673929 learned_addr learned my addr 172.21.15.146:0/2292673929 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/2292673929 >> v2:172.21.15.177:3300/0 conn(0x7f0bcc0d2aa0 0x7f0bcc06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bca7fc640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 0x7f0bcc0d2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3448482653 0 0) 0x7f0bcc0b3fc0 con 0x7f0bcc15cfd0 2024-04-25T15:20:41.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.351+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bb4003520 con 0x7f0bcc15cfd0 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1054596496 0 0) 0x7f0bb4003520 con 0x7f0bcc15cfd0 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 msgr2=0x7f0bcc0d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 0x7f0bcc0d2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 >> v2:172.21.15.177:3300/0 conn(0x7f0bcc0d2aa0 msgr2=0x7f0bcc06def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 --2- 172.21.15.146:0/2292673929 >> v2:172.21.15.177:3300/0 conn(0x7f0bcc0d2aa0 0x7f0bcc06def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc0cfff0 con 0x7f0bcc15cfd0 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f0bbc0019f0 con 0x7f0bcc15cfd0 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 >> v1:172.21.15.146:6789/0 conn(0x7f0bcc15cfd0 legacy=0x7f0bcc09bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb40039b0 0x7f0bb4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 0x7f0bb4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb4004d40 0x7f0bb4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cefe0 con 0x7f0bb4004d40 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc0d2eb0 con 0x7f0bb40039b0 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb400b9c0 con 0x7f0bcc0b3bb0 2024-04-25T15:20:41.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb4004d40 0x7f0bb4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcaffd640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb40039b0 0x7f0bb4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bca7fc640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 0x7f0bb4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 msgr2=0x7f0bb4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0b3bb0 0x7f0bb4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb40039b0 msgr2=0x7f0bb4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb40039b0 0x7f0bb4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.352+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/2292673929 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb4003520 con 0x7f0bb4004d40 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb4004d40 0x7f0bb4005150 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc09b460 tx=0x7f0bb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c737ff753ffd5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4036070 con 0x7f0bb4004d40 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc4031400 con 0x7f0bb4004d40 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bc9ffb640 1 -- 172.21.15.146:0/2292673929 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4009690 con 0x7f0bb4004d40 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bd1338640 1 -- 172.21.15.146:0/2292673929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb4004d40 msgr2=0x7f0bb4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bd1338640 1 --2- 172.21.15.146:0/2292673929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb4004d40 0x7f0bb4005150 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc09b460 tx=0x7f0bb4005830 comp rx=0 tx=0).stop 2024-04-25T15:20:41.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.353+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/2292673929 reap_dead start 2024-04-25T15:20:41.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.354+0000 7f0bd1338640 1 -- 172.21.15.146:0/2292673929 shutdown_connections 2024-04-25T15:20:41.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.354+0000 7f0bd1338640 1 -- 172.21.15.146:0/2292673929 >> 172.21.15.146:0/2292673929 conn(0x7f0bcc0b7530 msgr2=0x7f0bcc0b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:41.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.354+0000 7f0bd1338640 1 -- 172.21.15.146:0/2292673929 shutdown_connections 2024-04-25T15:20:41.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.354+0000 7f0bd1338640 1 -- 172.21.15.146:0/2292673929 wait complete. 2024-04-25T15:20:41.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.355+0000 7f0bd1338640 1 Processor -- start 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.356+0000 7f0bd1338640 1 -- start start 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bd1338640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc0cd7b0 0x7f0bcc0c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bd1338640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0cabf0 0x7f0bcc150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bd1338640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 0x7f0bcc0b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bd1338640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cf7f0 con 0x7f0bcc0b0930 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bd1338640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cf510 con 0x7f0bcc0cd7b0 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bd1338640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0bcc0cfad0 con 0x7f0bcc0cabf0 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 0x7f0bcc0b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 0x7f0bcc0b0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55904/0 (socket says 172.21.15.146:55904) 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/3314387115 learned_addr learned my addr 172.21.15.146:0/3314387115 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bca7fc640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0cabf0 0x7f0bcc150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcaffd640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc0cd7b0 0x7f0bcc0c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0cabf0 msgr2=0x7f0bcc150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0cabf0 0x7f0bcc150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc0cd7b0 msgr2=0x7f0bcc0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc0cd7b0 0x7f0bcc0c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 -- 172.21.15.146:0/3314387115 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc0b1280 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bca7fc640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0cabf0 0x7f0bcc150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcaffd640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc0cd7b0 0x7f0bcc0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bcb7fe640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 0x7f0bcc0b0d40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4034040 tx=0x7f0bc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.357+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4036070 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.358+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc4041070 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.358+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bcc0ce9a0 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.358+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4009690 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.358+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bcc0c9e30 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.359+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bcc160640 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.359+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0bc403b840 con 0x7f0bcc0b0930 2024-04-25T15:20:41.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.359+0000 7f0bb37fe640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bb80744b0 0x7f0bb80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.360+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f0bc40b5750 con 0x7f0bcc0b0930 2024-04-25T15:20:41.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.360+0000 7f0bcaffd640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bb80744b0 0x7f0bb80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.360+0000 7f0bcaffd640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bb80744b0 0x7f0bb80769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc004bc0 tx=0x7f0bbc006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.364+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bcc160640 con 0x7f0bcc0b0930 2024-04-25T15:20:41.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.541+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bcc06a0e0 con 0x7f0bcc0b0930 2024-04-25T15:20:41.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.542+0000 7f0bb37fe640 1 -- 172.21.15.146:0/3314387115 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f0bcc0b1280 con 0x7f0bcc0b0930 2024-04-25T15:20:41.544 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:41.544 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:41.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bb80744b0 msgr2=0x7f0bb80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bb80744b0 0x7f0bb80769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc004bc0 tx=0x7f0bbc006040 comp rx=0 tx=0).stop 2024-04-25T15:20:41.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 msgr2=0x7f0bcc0b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 0x7f0bcc0b0d40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4034040 tx=0x7f0bc4004730 comp rx=0 tx=0).stop 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 shutdown_connections 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bb80744b0 0x7f0bb80769c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bcc0b0930 0x7f0bcc0b0d40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bcc0cabf0 0x7f0bcc150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 --2- 172.21.15.146:0/3314387115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc0cd7b0 0x7f0bcc0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 >> 172.21.15.146:0/3314387115 conn(0x7f0bcc0b7530 msgr2=0x7f0bcc0c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 shutdown_connections 2024-04-25T15:20:41.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.544+0000 7f0bd1338640 1 -- 172.21.15.146:0/3314387115 wait complete. 2024-04-25T15:20:41.558 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:41.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:41.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.668+0000 7f7d69d9d640 1 Processor -- start 2024-04-25T15:20:41.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.668+0000 7f7d69d9d640 1 -- start start 2024-04-25T15:20:41.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d69d9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6415ff90 0x7f7d6406d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d69d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d6406d7c0 0x7f7d64163980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d69d9d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d640d5130 con 0x7f7d640d3e90 2024-04-25T15:20:41.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d69d9d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6406c550 con 0x7f7d6406d7c0 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d69d9d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6406c7a0 con 0x7f7d6415ff90 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d6406d7c0 0x7f7d64163980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d6406d7c0 0x7f7d64163980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55908/0 (socket says 172.21.15.146:55908) 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 -- 172.21.15.146:0/3880302322 learned_addr learned my addr 172.21.15.146:0/3880302322 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d63fff640 1 --2- 172.21.15.146:0/3880302322 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6415ff90 0x7f7d6406d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 -- 172.21.15.146:0/3880302322 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6415ff90 msgr2=0x7f7d6406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 --2- 172.21.15.146:0/3880302322 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6415ff90 0x7f7d6406d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 -- 172.21.15.146:0/3880302322 >> v1:172.21.15.146:6789/0 conn(0x7f7d640d3e90 legacy=0x7f7d640b8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.670+0000 7f7d6959c640 1 -- 172.21.15.146:0/3880302322 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6406ca80 con 0x7f7d6406d7c0 2024-04-25T15:20:41.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.671+0000 7f7d6959c640 1 --2- 172.21.15.146:0/3880302322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d6406d7c0 0x7f7d64163980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009e50 tx=0x7f7d5c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1beae61745d1032 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.671+0000 7f7d637fe640 1 -- 172.21.15.146:0/3880302322 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c036070 con 0x7f7d6406d7c0 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.671+0000 7f7d637fe640 1 -- 172.21.15.146:0/3880302322 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5c031600 con 0x7f7d6406d7c0 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.671+0000 7f7d637fe640 1 -- 172.21.15.146:0/3880302322 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c0319d0 con 0x7f7d6406d7c0 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 -- 172.21.15.146:0/3880302322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d6406d7c0 msgr2=0x7f7d64163980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/3880302322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d6406d7c0 0x7f7d64163980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009e50 tx=0x7f7d5c02a5f0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 -- 172.21.15.146:0/3880302322 shutdown_connections 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/3880302322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d6406d7c0 0x7f7d64163980 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/3880302322 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6415ff90 0x7f7d6406d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 -- 172.21.15.146:0/3880302322 >> 172.21.15.146:0/3880302322 conn(0x7f7d640d6490 msgr2=0x7f7d640d2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 -- 172.21.15.146:0/3880302322 shutdown_connections 2024-04-25T15:20:41.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.672+0000 7f7d69d9d640 1 -- 172.21.15.146:0/3880302322 wait complete. 2024-04-25T15:20:41.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.673+0000 7f7d69d9d640 1 Processor -- start 2024-04-25T15:20:41.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.674+0000 7f7d69d9d640 1 -- start start 2024-04-25T15:20:41.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d69d9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6406d7c0 0x7f7d64150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d69d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 0x7f7d641549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d69d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d6415ff90 0x7f7d64163970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d69d9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7d640d4d80 con 0x7f7d640d3e90 2024-04-25T15:20:41.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d69d9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6406dbd0 con 0x7f7d6415ff90 2024-04-25T15:20:41.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d69d9d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7d641619e0 con 0x7f7d6406d7c0 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 0x7f7d641549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 0x7f7d641549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55910/0 (socket says 172.21.15.146:55910) 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 -- 172.21.15.146:0/2066157589 learned_addr learned my addr 172.21.15.146:0/2066157589 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d6959c640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d6415ff90 0x7f7d64163970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d68d9b640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6406d7c0 0x7f7d64150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 -- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6406d7c0 msgr2=0x7f7d64150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6406d7c0 0x7f7d64150750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 -- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d6415ff90 msgr2=0x7f7d64163970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d6415ff90 0x7f7d64163970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.675+0000 7f7d63fff640 1 -- 172.21.15.146:0/2066157589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d641f6ed0 con 0x7f7d640d3e90 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.676+0000 7f7d63fff640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 0x7f7d641549e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d5f0 tx=0x7f7d5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.676+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d580125f0 con 0x7f7d640d3e90 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.676+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58019070 con 0x7f7d640d3e90 2024-04-25T15:20:41.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.676+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d641f6350 con 0x7f7d640d3e90 2024-04-25T15:20:41.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.676+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d580129d0 con 0x7f7d640d3e90 2024-04-25T15:20:41.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.676+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d641f6910 con 0x7f7d640d3e90 2024-04-25T15:20:41.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.677+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d64159ea0 con 0x7f7d640d3e90 2024-04-25T15:20:41.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.678+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7d58012b70 con 0x7f7d640d3e90 2024-04-25T15:20:41.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.678+0000 7f7d617fa640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7d500744e0 0x7f7d500769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.678+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f7d580959b0 con 0x7f7d640d3e90 2024-04-25T15:20:41.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.678+0000 7f7d68d9b640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7d500744e0 0x7f7d500769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.678+0000 7f7d68d9b640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7d500744e0 0x7f7d500769f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d54007fd0 tx=0x7f7d54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.683+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d5805fde0 con 0x7f7d640d3e90 2024-04-25T15:20:41.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.858+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d641f60b0 con 0x7f7d640d3e90 2024-04-25T15:20:41.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.859+0000 7f7d617fa640 1 -- 172.21.15.146:0/2066157589 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7d5805f6e0 con 0x7f7d640d3e90 2024-04-25T15:20:41.861 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.860+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7d500744e0 msgr2=0x7f7d500769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.860+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7d500744e0 0x7f7d500769f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d54007fd0 tx=0x7f7d54009040 comp rx=0 tx=0).stop 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 msgr2=0x7f7d641549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 0x7f7d641549e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d5f0 tx=0x7f7d5800dab0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 shutdown_connections 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7d500744e0 0x7f7d500769f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d6415ff90 0x7f7d64163970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d640d3e90 0x7f7d641549e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 --2- 172.21.15.146:0/2066157589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d6406d7c0 0x7f7d64150750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 >> 172.21.15.146:0/2066157589 conn(0x7f7d640d6490 msgr2=0x7f7d6415c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 shutdown_connections 2024-04-25T15:20:41.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.861+0000 7f7d69d9d640 1 -- 172.21.15.146:0/2066157589 wait complete. 2024-04-25T15:20:41.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:41.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.983+0000 7fc004f72640 1 Processor -- start 2024-04-25T15:20:41.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.983+0000 7fc004f72640 1 -- start start 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fc004f72640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc0000d2d90 0x7fc00009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fc004f72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0000b1960 con 0x7fc0000b2400 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fc004f72640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0000b3ce0 con 0x7fc0000ccd40 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fc004f72640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc0000b1c00 con 0x7fc0000d2d90 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc0000b2400 0x7fc0000b2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34636/0 (socket says 172.21.15.146:34636) 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffed9d640 1 -- 172.21.15.146:0/123788597 learned_addr learned my addr 172.21.15.146:0/123788597 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918309280 0 0) 0x7fc0000b1960 con 0x7fc0000b2400 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffdd9b640 1 --2- 172.21.15.146:0/123788597 >> v2:172.21.15.177:3300/0 conn(0x7fc0000d2d90 0x7fc00009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe8003520 con 0x7fc0000b2400 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3665233239 0 0) 0x7fbfe8003520 con 0x7fc0000b2400 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 >> v2:172.21.15.177:3300/0 conn(0x7fc0000d2d90 msgr2=0x7fc00009bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 --2- 172.21.15.146:0/123788597 >> v2:172.21.15.177:3300/0 conn(0x7fc0000d2d90 0x7fc00009bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 >> v1:172.21.15.177:6789/0 conn(0x7fc0000ccd40 legacy=0x7fc0000d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0000b1ee0 con 0x7fc0000b2400 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.985+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fbff8001690 con 0x7fc0000b2400 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 >> v1:172.21.15.146:6789/0 conn(0x7fc0000b2400 legacy=0x7fc0000b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfe80039b0 0x7fbfe8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe80051c0 0x7fbfe800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0000b1c00 con 0x7fbfe80051c0 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe800c160 con 0x7fbfe8004800 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe800c360 con 0x7fbfe80039b0 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffed9d640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe80051c0 0x7fbfe800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffdd9b640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffe59c640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfe80039b0 0x7fbfe8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffed9d640 1 -- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfe80039b0 msgr2=0x7fbfe8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffed9d640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfe80039b0 0x7fbfe8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffed9d640 1 -- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe8004800 msgr2=0x7fbfe8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffed9d640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe8004800 0x7fbfe8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.986+0000 7fbffed9d640 1 -- 172.21.15.146:0/123788597 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe8003520 con 0x7fbfe80051c0 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fbffed9d640 1 -- 172.21.15.146:0/123788597 reap_dead start 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fbffed9d640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe80051c0 0x7fbfe800b9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c210 tx=0x7fbfe8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c667b0146916bf6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8007d10 con 0x7fbfe80051c0 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff8005ad0 con 0x7fbfe80051c0 2024-04-25T15:20:41.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fbffd59a640 1 -- 172.21.15.146:0/123788597 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8007070 con 0x7fbfe80051c0 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fc004f72640 1 -- 172.21.15.146:0/123788597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe80051c0 msgr2=0x7fbfe800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.987+0000 7fc004f72640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe80051c0 0x7fbfe800b9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c210 tx=0x7fbfe8005830 comp rx=0 tx=0).stop 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.988+0000 7fc004f72640 1 -- 172.21.15.146:0/123788597 shutdown_connections 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.988+0000 7fc004f72640 1 --2- 172.21.15.146:0/123788597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe80051c0 0x7fbfe800b9c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.988+0000 7fc004f72640 1 -- 172.21.15.146:0/123788597 >> 172.21.15.146:0/123788597 conn(0x7fc0000b7400 msgr2=0x7fc0000b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.988+0000 7fc004f72640 1 -- 172.21.15.146:0/123788597 shutdown_connections 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.988+0000 7fc004f72640 1 -- 172.21.15.146:0/123788597 wait complete. 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.989+0000 7fc004f72640 1 Processor -- start 2024-04-25T15:20:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.990+0000 7fc004f72640 1 -- start start 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fc004f72640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe80051c0 0x7fc0000c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fc004f72640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 0x7fc00006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fc004f72640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc00006d320 0x7fc0001f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fc004f72640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0000b3ac0 con 0x7fc00006e4a0 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fc004f72640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc0000b3940 con 0x7fbfe80051c0 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fc004f72640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc00009b440 con 0x7fc00006d320 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 0x7fc00006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 0x7fc00006cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55932/0 (socket says 172.21.15.146:55932) 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 -- 172.21.15.146:0/4074607422 learned_addr learned my addr 172.21.15.146:0/4074607422 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffe59c640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe80051c0 0x7fc0000c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffed9d640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc00006d320 0x7fc0001f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 -- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc00006d320 msgr2=0x7fc0001f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc00006d320 0x7fc0001f97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 -- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe80051c0 msgr2=0x7fc0000c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:41.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe80051c0 0x7fc0000c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.991+0000 7fbffdd9b640 1 -- 172.21.15.146:0/4074607422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0000c6900 con 0x7fc00006e4a0 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.992+0000 7fbffdd9b640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 0x7fc00006cde0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbff4011990 tx=0x7fbff4011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.992+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4019070 con 0x7fc00006e4a0 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.992+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff4004300 con 0x7fc00006e4a0 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.992+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff400fa80 con 0x7fc00006e4a0 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.992+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0000c6be0 con 0x7fc00006e4a0 2024-04-25T15:20:41.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.992+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0000c71a0 con 0x7fc00006e4a0 2024-04-25T15:20:41.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.993+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc000163290 con 0x7fc00006e4a0 2024-04-25T15:20:41.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.993+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbff401c070 con 0x7fc00006e4a0 2024-04-25T15:20:41.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.994+0000 7fbfeeffd640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbfe00744e0 0x7fbfe00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:41.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.994+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fbff4095a30 con 0x7fc00006e4a0 2024-04-25T15:20:41.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.994+0000 7fbffe59c640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbfe00744e0 0x7fbfe00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:41.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.994+0000 7fbffe59c640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbfe00744e0 0x7fbfe00769f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000f80 tx=0x7fbff0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:41.999+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbff405fdd0 con 0x7fc00006e4a0 2024-04-25T15:20:42.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.175+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc00006a0e0 con 0x7fc00006e4a0 2024-04-25T15:20:42.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.175+0000 7fbfeeffd640 1 -- 172.21.15.146:0/4074607422 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fbff405f6d0 con 0x7fc00006e4a0 2024-04-25T15:20:42.177 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:42.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbfe00744e0 msgr2=0x7fbfe00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbfe00744e0 0x7fbfe00769f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000f80 tx=0x7fbff0009040 comp rx=0 tx=0).stop 2024-04-25T15:20:42.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 msgr2=0x7fc00006cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 0x7fc00006cde0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbff4011990 tx=0x7fbff4011e50 comp rx=0 tx=0).stop 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 shutdown_connections 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbfe00744e0 0x7fbfe00769f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc00006d320 0x7fc0001f97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00006e4a0 0x7fc00006cde0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 --2- 172.21.15.146:0/4074607422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfe80051c0 0x7fc0000c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 >> 172.21.15.146:0/4074607422 conn(0x7fc0000b7400 msgr2=0x7fc00015f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 shutdown_connections 2024-04-25T15:20:42.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.177+0000 7fc004f72640 1 -- 172.21.15.146:0/4074607422 wait complete. 2024-04-25T15:20:42.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:42.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.300+0000 7fda52f38640 1 Processor -- start 2024-04-25T15:20:42.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.300+0000 7fda52f38640 1 -- start start 2024-04-25T15:20:42.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda52f38640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fda4c0d23d0 0x7fda4c08cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda52f38640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4c0ce5a0 con 0x7fda4c15e180 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda52f38640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4c08d450 con 0x7fda4c08d040 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda52f38640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fda4c0ce840 con 0x7fda4c0d23d0 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda50cd4640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fda4c15e180 0x7fda4c0cf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34648/0 (socket says 172.21.15.146:34648) 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda50cd4640 1 -- 172.21.15.146:0/894542340 learned_addr learned my addr 172.21.15.146:0/894542340 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda4bfff640 1 --2- 172.21.15.146:0/894542340 >> v2:172.21.15.177:3300/0 conn(0x7fda4c0d23d0 0x7fda4c08cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda4b7fe640 1 -- 172.21.15.146:0/894542340 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506712182 0 0) 0x7fda4c0ce5a0 con 0x7fda4c15e180 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.301+0000 7fda4b7fe640 1 -- 172.21.15.146:0/894542340 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda38003520 con 0x7fda4c15e180 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.302+0000 7fda4b7fe640 1 -- 172.21.15.146:0/894542340 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2401243516 0 0) 0x7fda38003520 con 0x7fda4c15e180 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.302+0000 7fda4bfff640 1 -- 172.21.15.146:0/894542340 >> v1:172.21.15.177:6789/0 conn(0x7fda4c08d040 legacy=0x7fda4c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.302+0000 7fda4bfff640 1 -- 172.21.15.146:0/894542340 >> v1:172.21.15.146:6789/0 conn(0x7fda4c15e180 legacy=0x7fda4c0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.302+0000 7fda4bfff640 1 -- 172.21.15.146:0/894542340 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c0ceb20 con 0x7fda4c0d23d0 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.302+0000 7fda4bfff640 1 --2- 172.21.15.146:0/894542340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c0d23d0 0x7fda4c08cb00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda340097d0 tx=0x7fda3402a200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4817b424780a1099 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda4b7fe640 1 -- 172.21.15.146:0/894542340 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34037070 con 0x7fda4c0d23d0 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda4b7fe640 1 -- 172.21.15.146:0/894542340 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3402ac90 con 0x7fda4c0d23d0 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda4b7fe640 1 -- 172.21.15.146:0/894542340 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c0017f0 con 0x7fda4c0d23d0 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 -- 172.21.15.146:0/894542340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c0d23d0 msgr2=0x7fda4c08cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 --2- 172.21.15.146:0/894542340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c0d23d0 0x7fda4c08cb00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fda340097d0 tx=0x7fda3402a200 comp rx=0 tx=0).stop 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 -- 172.21.15.146:0/894542340 shutdown_connections 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 --2- 172.21.15.146:0/894542340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c0d23d0 0x7fda4c08cb00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 -- 172.21.15.146:0/894542340 >> 172.21.15.146:0/894542340 conn(0x7fda4c0b52e0 msgr2=0x7fda4c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:42.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 -- 172.21.15.146:0/894542340 shutdown_connections 2024-04-25T15:20:42.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.303+0000 7fda52f38640 1 -- 172.21.15.146:0/894542340 wait complete. 2024-04-25T15:20:42.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.305+0000 7fda52f38640 1 Processor -- start 2024-04-25T15:20:42.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.305+0000 7fda52f38640 1 -- start start 2024-04-25T15:20:42.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c08d040 0x7fda4c0caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda4c0d23d0 0x7fda4c0c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 0x7fda4c15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0cf3c0 con 0x7fda4c15e180 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0cf0e0 con 0x7fda4c08d040 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c15e590 con 0x7fda4c0d23d0 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 0x7fda4c15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 0x7fda4c15b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55934/0 (socket says 172.21.15.146:55934) 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 -- 172.21.15.146:0/3171287325 learned_addr learned my addr 172.21.15.146:0/3171287325 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda4bfff640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda4c0d23d0 0x7fda4c0c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda50cd4640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c08d040 0x7fda4c0caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 -- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda4c0d23d0 msgr2=0x7fda4c0c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda4c0d23d0 0x7fda4c0c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 -- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c08d040 msgr2=0x7fda4c0caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c08d040 0x7fda4c0caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 -- 172.21.15.146:0/3171287325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c0c9a70 con 0x7fda4c15e180 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda514d5640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 0x7fda4c15b090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c760 tx=0x7fda4400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44003b70 con 0x7fda4c15e180 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda44003d10 con 0x7fda4c15e180 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4400aa90 con 0x7fda4c15e180 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c1604c0 con 0x7fda4c15e180 2024-04-25T15:20:42.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.307+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c15ebd0 con 0x7fda4c15e180 2024-04-25T15:20:42.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.309+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c06a0e0 con 0x7fda4c15e180 2024-04-25T15:20:42.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.309+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fda4400ac30 con 0x7fda4c15e180 2024-04-25T15:20:42.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.310+0000 7fda497fa640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda2c0744e0 0x7fda2c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.310+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fda440954a0 con 0x7fda4c15e180 2024-04-25T15:20:42.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.310+0000 7fda50cd4640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda2c0744e0 0x7fda2c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.310+0000 7fda50cd4640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda2c0744e0 0x7fda2c0769f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda3c001000 tx=0x7fda3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.314+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda4c0c9a70 con 0x7fda4c15e180 2024-04-25T15:20:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.494+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fda4c0b5a70 con 0x7fda4c15e180 2024-04-25T15:20:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.494+0000 7fda497fa640 1 -- 172.21.15.146:0/3171287325 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fda4405f840 con 0x7fda4c15e180 2024-04-25T15:20:42.496 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda2c0744e0 msgr2=0x7fda2c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda2c0744e0 0x7fda2c0769f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda3c001000 tx=0x7fda3c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 msgr2=0x7fda4c15b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 0x7fda4c15b090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c760 tx=0x7fda4400cc20 comp rx=0 tx=0).stop 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 shutdown_connections 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fda2c0744e0 0x7fda2c0769f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda4c15e180 0x7fda4c15b090 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda4c0d23d0 0x7fda4c0c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 --2- 172.21.15.146:0/3171287325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda4c08d040 0x7fda4c0caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 >> 172.21.15.146:0/3171287325 conn(0x7fda4c0b52e0 msgr2=0x7fda4c0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 shutdown_connections 2024-04-25T15:20:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.497+0000 7fda52f38640 1 -- 172.21.15.146:0/3171287325 wait complete. 2024-04-25T15:20:42.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:42.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.620+0000 7fbf491c5640 1 Processor -- start 2024-04-25T15:20:42.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.620+0000 7fbf491c5640 1 -- start start 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf491c5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf44163c00 0x7fbf4406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf491c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf440d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf491c5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf4406dea0 con 0x7fbf440d0de0 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf491c5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbf440b5230 con 0x7fbf44163c00 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf491c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbf4406c8c0 con 0x7fbf440bb450 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf42d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf44163c00 0x7fbf4406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf4359e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbf440d0de0 0x7fbf440b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34658/0 (socket says 172.21.15.146:34658) 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf42d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf44163c00 0x7fbf4406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55948/0 (socket says 172.21.15.146:55948) 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf4359e640 1 -- 172.21.15.146:0/602196168 learned_addr learned my addr 172.21.15.146:0/602196168 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf4259c640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf440d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf42d9d640 1 -- 172.21.15.146:0/602196168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 msgr2=0x7fbf440d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf42d9d640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf440d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf42d9d640 1 -- 172.21.15.146:0/602196168 >> v1:172.21.15.146:6789/0 conn(0x7fbf440d0de0 legacy=0x7fbf440b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf41d9b640 1 -- 172.21.15.146:0/602196168 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2120753521 0 0) 0x7fbf4406dea0 con 0x7fbf440d0de0 2024-04-25T15:20:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf42d9d640 1 -- 172.21.15.146:0/602196168 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4406cba0 con 0x7fbf44163c00 2024-04-25T15:20:42.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.621+0000 7fbf4259c640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf440d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:42.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.622+0000 7fbf42d9d640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4406d960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf3402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebed008479f4cce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.622+0000 7fbf41d9b640 1 -- 172.21.15.146:0/602196168 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34031600 con 0x7fbf44163c00 2024-04-25T15:20:42.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.622+0000 7fbf41d9b640 1 -- 172.21.15.146:0/602196168 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf340317a0 con 0x7fbf44163c00 2024-04-25T15:20:42.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.622+0000 7fbf41d9b640 1 -- 172.21.15.146:0/602196168 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34031b70 con 0x7fbf44163c00 2024-04-25T15:20:42.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 -- 172.21.15.146:0/602196168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 msgr2=0x7fbf4406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4406d960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009d90 tx=0x7fbf3402a5d0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 -- 172.21.15.146:0/602196168 shutdown_connections 2024-04-25T15:20:42.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf440d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 --2- 172.21.15.146:0/602196168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4406d960 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 -- 172.21.15.146:0/602196168 >> 172.21.15.146:0/602196168 conn(0x7fbf440b7970 msgr2=0x7fbf440b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 -- 172.21.15.146:0/602196168 shutdown_connections 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.623+0000 7fbf491c5640 1 -- 172.21.15.146:0/602196168 wait complete. 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.625+0000 7fbf491c5640 1 Processor -- start 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.625+0000 7fbf491c5640 1 -- start start 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.626+0000 7fbf491c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf4414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.626+0000 7fbf491c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf440d0de0 0x7fbf4415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.626+0000 7fbf491c5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.626+0000 7fbf491c5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4406c550 con 0x7fbf44163c00 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.626+0000 7fbf491c5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4406d160 con 0x7fbf440d0de0 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.626+0000 7fbf491c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbf440bb860 con 0x7fbf440bb450 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4414b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55960/0 (socket says 172.21.15.146:55960) 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 -- 172.21.15.146:0/1553457829 learned_addr learned my addr 172.21.15.146:0/1553457829 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf42d9d640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf4414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4259c640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf440d0de0 0x7fbf4415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 -- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 msgr2=0x7fbf4414dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf4414dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 -- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf440d0de0 msgr2=0x7fbf4415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf440d0de0 0x7fbf4415ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 -- 172.21.15.146:0/1553457829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4415b080 con 0x7fbf44163c00 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4259c640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf440d0de0 0x7fbf4415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf4359e640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4414b670 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c3e0 tx=0x7fbf3c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0074e0 con 0x7fbf44163c00 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3c0047e0 con 0x7fbf44163c00 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c004bd0 con 0x7fbf44163c00 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf44159d80 con 0x7fbf44163c00 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.627+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4415a340 con 0x7fbf44163c00 2024-04-25T15:20:42.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.629+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4406a0e0 con 0x7fbf44163c00 2024-04-25T15:20:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.629+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbf3c011070 con 0x7fbf44163c00 2024-04-25T15:20:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.630+0000 7fbf2b7fe640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf300744b0 0x7fbf300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.630+0000 7fbf42d9d640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf300744b0 0x7fbf300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.630+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fbf3c095090 con 0x7fbf44163c00 2024-04-25T15:20:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.630+0000 7fbf42d9d640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf300744b0 0x7fbf300769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbf34034000 tx=0x7fbf3402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.634+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf3c05f440 con 0x7fbf44163c00 2024-04-25T15:20:42.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.813+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbf440d3ed0 con 0x7fbf44163c00 2024-04-25T15:20:42.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.814+0000 7fbf2b7fe640 1 -- 172.21.15.146:0/1553457829 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fbf3c067430 con 0x7fbf44163c00 2024-04-25T15:20:42.816 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf300744b0 msgr2=0x7fbf300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf300744b0 0x7fbf300769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbf34034000 tx=0x7fbf3402f000 comp rx=0 tx=0).stop 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 msgr2=0x7fbf4414b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4414b670 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c3e0 tx=0x7fbf3c00c8a0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 shutdown_connections 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbf300744b0 0x7fbf300769c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf44163c00 0x7fbf4414b670 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf440d0de0 0x7fbf4415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 --2- 172.21.15.146:0/1553457829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf440bb450 0x7fbf4414dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 >> 172.21.15.146:0/1553457829 conn(0x7fbf440b7970 msgr2=0x7fbf440cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:42.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 shutdown_connections 2024-04-25T15:20:42.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.816+0000 7fbf491c5640 1 -- 172.21.15.146:0/1553457829 wait complete. 2024-04-25T15:20:42.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.938+0000 7f99b3697640 1 Processor -- start 2024-04-25T15:20:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.939+0000 7f99b3697640 1 -- start start 2024-04-25T15:20:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b3697640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99ac0b2ba0 0x7f99ac1645c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b3697640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b3697640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99ac0b8eb0 con 0x7f99ac06ee60 2024-04-25T15:20:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b3697640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f99ac161840 con 0x7f99ac0b2ba0 2024-04-25T15:20:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b3697640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f99ac08d730 con 0x7f99ac164b00 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99ac0b2ba0 0x7f99ac1645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99ac0b2ba0 0x7f99ac1645c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55976/0 (socket says 172.21.15.146:55976) 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 -- 172.21.15.146:0/3754662438 learned_addr learned my addr 172.21.15.146:0/3754662438 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 -- 172.21.15.146:0/3754662438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 msgr2=0x7f99ac0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b1c34640 1 --2- 172.21.15.146:0/3754662438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 --2- 172.21.15.146:0/3754662438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 -- 172.21.15.146:0/3754662438 >> v1:172.21.15.146:6789/0 conn(0x7f99ac06ee60 legacy=0x7f99ac163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.940+0000 7f99b0c32640 1 -- 172.21.15.146:0/3754662438 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ac08ce00 con 0x7f99ac0b2ba0 2024-04-25T15:20:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.941+0000 7f99b0c32640 1 --2- 172.21.15.146:0/3754662438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac1645c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f999c009870 tx=0x7f999c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bbd523fa5f6f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.941+0000 7f99a3fff640 1 -- 172.21.15.146:0/3754662438 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c037070 con 0x7f99ac0b2ba0 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.941+0000 7f99a3fff640 1 -- 172.21.15.146:0/3754662438 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f999c032440 con 0x7f99ac0b2ba0 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.941+0000 7f99a3fff640 1 -- 172.21.15.146:0/3754662438 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c032810 con 0x7f99ac0b2ba0 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 -- 172.21.15.146:0/3754662438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac0b2ba0 msgr2=0x7f99ac1645c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 --2- 172.21.15.146:0/3754662438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac1645c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f999c009870 tx=0x7f999c02a270 comp rx=0 tx=0).stop 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 -- 172.21.15.146:0/3754662438 shutdown_connections 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 --2- 172.21.15.146:0/3754662438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 --2- 172.21.15.146:0/3754662438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac1645c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 -- 172.21.15.146:0/3754662438 >> 172.21.15.146:0/3754662438 conn(0x7f99ac0b52e0 msgr2=0x7f99ac0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 -- 172.21.15.146:0/3754662438 shutdown_connections 2024-04-25T15:20:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.942+0000 7f99b3697640 1 -- 172.21.15.146:0/3754662438 wait complete. 2024-04-25T15:20:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.944+0000 7f99b3697640 1 Processor -- start 2024-04-25T15:20:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.944+0000 7f99b3697640 1 -- start start 2024-04-25T15:20:42.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b3697640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 0x7f99ac0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b3697640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac14df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b3697640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0728a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b3697640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac08d3c0 con 0x7f99ac06ee60 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b3697640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac06f440 con 0x7f99ac0b2ba0 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b3697640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f99ac08dbc0 con 0x7f99ac164b00 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b1433640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 0x7f99ac0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b1433640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 0x7f99ac0b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55984/0 (socket says 172.21.15.146:55984) 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b1433640 1 -- 172.21.15.146:0/3575446534 learned_addr learned my addr 172.21.15.146:0/3575446534 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.945+0000 7f99b1c34640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b0c32640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac14df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b1433640 1 -- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 msgr2=0x7f99ac0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b1433640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0728a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b1433640 1 -- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99ac0b2ba0 msgr2=0x7f99ac14df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b1433640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac14df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b1433640 1 -- 172.21.15.146:0/3575446534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ac14ebb0 con 0x7f99ac06ee60 2024-04-25T15:20:42.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b1433640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 0x7f99ac0b3d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c760 tx=0x7f99a800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8017070 con 0x7f99ac06ee60 2024-04-25T15:20:42.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a8003da0 con 0x7f99ac06ee60 2024-04-25T15:20:42.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ac14ee90 con 0x7f99ac06ee60 2024-04-25T15:20:42.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8005290 con 0x7f99ac06ee60 2024-04-25T15:20:42.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.946+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ac14f450 con 0x7f99ac06ee60 2024-04-25T15:20:42.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.947+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ac06a0e0 con 0x7f99ac06ee60 2024-04-25T15:20:42.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.948+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f99a801a070 con 0x7f99ac06ee60 2024-04-25T15:20:42.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.948+0000 7f99a1ffb640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99900744b0 0x7f99900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:42.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.948+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f99a8094bc0 con 0x7f99ac06ee60 2024-04-25T15:20:42.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.948+0000 7f99b0c32640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99900744b0 0x7f99900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:42.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.949+0000 7f99b0c32640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99900744b0 0x7f99900769c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f999c034000 tx=0x7f999c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:42.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:42.953+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99a805ef60 con 0x7f99ac06ee60 2024-04-25T15:20:43.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.138+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f99ac0b5fb0 con 0x7f99ac06ee60 2024-04-25T15:20:43.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.138+0000 7f99a1ffb640 1 -- 172.21.15.146:0/3575446534 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f99a805e860 con 0x7f99ac06ee60 2024-04-25T15:20:43.140 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:43.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99900744b0 msgr2=0x7f99900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99900744b0 0x7f99900769c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f999c034000 tx=0x7f999c02e000 comp rx=0 tx=0).stop 2024-04-25T15:20:43.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 msgr2=0x7f99ac0b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 0x7f99ac0b3d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f99a800c760 tx=0x7f99a800cc20 comp rx=0 tx=0).stop 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 shutdown_connections 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99900744b0 0x7f99900769c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99ac164b00 0x7f99ac0728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99ac0b2ba0 0x7f99ac14df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 --2- 172.21.15.146:0/3575446534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99ac06ee60 0x7f99ac0b3d90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 >> 172.21.15.146:0/3575446534 conn(0x7f99ac0b52e0 msgr2=0x7f99ac0ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 shutdown_connections 2024-04-25T15:20:43.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.140+0000 7f99b3697640 1 -- 172.21.15.146:0/3575446534 wait complete. 2024-04-25T15:20:43.153 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:20:43.154 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:43.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:43.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.280+0000 7f5bf6c30640 1 Processor -- start 2024-04-25T15:20:43.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.280+0000 7f5bf6c30640 1 -- start start 2024-04-25T15:20:43.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf6c30640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5bf015ded0 0x7f5bf00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf6c30640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf00b3c10 0x7f5bf00b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf6c30640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bf009c2a0 con 0x7f5bf00d21f0 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf6c30640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf00d1d60 con 0x7f5bf00b3c10 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf6c30640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf00b4020 con 0x7f5bf015ded0 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf51cd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf00b3c10 0x7f5bf00b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf51cd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf00b3c10 0x7f5bf00b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55988/0 (socket says 172.21.15.146:55988) 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3955158827 learned_addr learned my addr 172.21.15.146:0/3955158827 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.281+0000 7f5beffff640 1 --2- 172.21.15.146:0/3955158827 >> v2:172.21.15.177:3300/0 conn(0x7f5bf015ded0 0x7f5bf00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3955158827 >> v2:172.21.15.177:3300/0 conn(0x7f5bf015ded0 msgr2=0x7f5bf00b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bf51cd640 1 --2- 172.21.15.146:0/3955158827 >> v2:172.21.15.177:3300/0 conn(0x7f5bf015ded0 0x7f5bf00b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3955158827 >> v1:172.21.15.177:6789/0 conn(0x7f5bf00d21f0 legacy=0x7f5bf006f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3955158827 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf009bce0 con 0x7f5bf00b3c10 2024-04-25T15:20:43.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bf51cd640 1 --2- 172.21.15.146:0/3955158827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00b3c10 0x7f5bf00b21b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5bf009ba30 tx=0x7f5be802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3771cdf7c34c2ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bef7fe640 1 -- 172.21.15.146:0/3955158827 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be8036070 con 0x7f5bf00b3c10 2024-04-25T15:20:43.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bef7fe640 1 -- 172.21.15.146:0/3955158827 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be8031600 con 0x7f5bf00b3c10 2024-04-25T15:20:43.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.282+0000 7f5bef7fe640 1 -- 172.21.15.146:0/3955158827 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be80319f0 con 0x7f5bf00b3c10 2024-04-25T15:20:43.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3955158827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00b3c10 msgr2=0x7f5bf00b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3955158827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00b3c10 0x7f5bf00b21b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5bf009ba30 tx=0x7f5be802a7c0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3955158827 shutdown_connections 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3955158827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00b3c10 0x7f5bf00b21b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3955158827 >> v2:172.21.15.177:3300/0 conn(0x7f5bf015ded0 0x7f5bf00b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3955158827 >> 172.21.15.146:0/3955158827 conn(0x7f5bf00b7530 msgr2=0x7f5bf00b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3955158827 shutdown_connections 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.283+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3955158827 wait complete. 2024-04-25T15:20:43.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.285+0000 7f5bf6c30640 1 Processor -- start 2024-04-25T15:20:43.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.285+0000 7f5bf6c30640 1 -- start start 2024-04-25T15:20:43.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf6c30640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bf00b3c10 0x7f5bf015fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf6c30640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf00d21f0 0x7f5bf0150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf6c30640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 0x7f5bf015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf6c30640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf015e2e0 con 0x7f5bf015ded0 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf6c30640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf00d19f0 con 0x7f5bf00d21f0 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf6c30640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf009b400 con 0x7f5bf00b3c10 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf51cd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 0x7f5bf015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf51cd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 0x7f5bf015c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55990/0 (socket says 172.21.15.146:55990) 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.286+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3213078233 learned_addr learned my addr 172.21.15.146:0/3213078233 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf49cc640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bf00b3c10 0x7f5bf015fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5beffff640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf00d21f0 0x7f5bf0150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bf00b3c10 msgr2=0x7f5bf015fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf51cd640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bf00b3c10 0x7f5bf015fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf00d21f0 msgr2=0x7f5bf0150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf51cd640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf00d21f0 0x7f5bf0150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf51cd640 1 -- 172.21.15.146:0/3213078233 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf0160d20 con 0x7f5bf015ded0 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf49cc640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bf00b3c10 0x7f5bf015fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:43.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf51cd640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 0x7f5bf015c460 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5be802adb0 tx=0x7f5be8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be8036050 con 0x7f5bf015ded0 2024-04-25T15:20:43.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be8037040 con 0x7f5bf015ded0 2024-04-25T15:20:43.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf01f84e0 con 0x7f5bf015ded0 2024-04-25T15:20:43.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be8009510 con 0x7f5bf015ded0 2024-04-25T15:20:43.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.287+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf01f7a00 con 0x7f5bf015ded0 2024-04-25T15:20:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.288+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf006a0e0 con 0x7f5bf015ded0 2024-04-25T15:20:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.289+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5be8044020 con 0x7f5bf015ded0 2024-04-25T15:20:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.289+0000 7f5bed7fa640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bd00744b0 0x7f5bd00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.289+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5be80b59e0 con 0x7f5bf015ded0 2024-04-25T15:20:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.289+0000 7f5bf49cc640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bd00744b0 0x7f5bd00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.290+0000 7f5bf49cc640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bd00744b0 0x7f5bd00769c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80015e0 tx=0x7f5bd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.294+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5be804b020 con 0x7f5bf015ded0 2024-04-25T15:20:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.470+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5bf015ac20 con 0x7f5bf015ded0 2024-04-25T15:20:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.471+0000 7f5bed7fa640 1 -- 172.21.15.146:0/3213078233 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f5be807fe10 con 0x7f5bf015ded0 2024-04-25T15:20:43.473 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:43.473 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bd00744b0 msgr2=0x7f5bd00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bd00744b0 0x7f5bd00769c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80015e0 tx=0x7f5bd8009040 comp rx=0 tx=0).stop 2024-04-25T15:20:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 msgr2=0x7f5bf015c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 0x7f5bf015c460 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5be802adb0 tx=0x7f5be8004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 shutdown_connections 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bd00744b0 0x7f5bd00769c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015ded0 0x7f5bf015c460 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf00d21f0 0x7f5bf0150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 --2- 172.21.15.146:0/3213078233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bf00b3c10 0x7f5bf015fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 >> 172.21.15.146:0/3213078233 conn(0x7f5bf00b7530 msgr2=0x7f5bf00b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.473+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 shutdown_connections 2024-04-25T15:20:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.474+0000 7f5bf6c30640 1 -- 172.21.15.146:0/3213078233 wait complete. 2024-04-25T15:20:43.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:43.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:43.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.596+0000 7fc39c630640 1 Processor -- start 2024-04-25T15:20:43.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.596+0000 7fc39c630640 1 -- start start 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc39c630640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc3940b4cc0 0x7fc39406f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc39c630640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39408dd40 con 0x7fc39406fc20 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc39c630640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3940d27b0 con 0x7fc3940b6030 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc39c630640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc39408e020 con 0x7fc3940b4cc0 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc39abcd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc39406fc20 0x7fc3940cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34672/0 (socket says 172.21.15.146:34672) 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc39abcd640 1 -- 172.21.15.146:0/2191698694 learned_addr learned my addr 172.21.15.146:0/2191698694 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:43.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.597+0000 7fc399bcb640 1 --2- 172.21.15.146:0/2191698694 >> v2:172.21.15.177:3300/0 conn(0x7fc3940b4cc0 0x7fc39406f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581529360 0 0) 0x7fc39408dd40 con 0x7fc39406fc20 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc380003520 con 0x7fc39406fc20 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279202990 0 0) 0x7fc3940d27b0 con 0x7fc3940b6030 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc39408dd40 con 0x7fc3940b6030 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2792949475 0 0) 0x7fc380003520 con 0x7fc39406fc20 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 >> v2:172.21.15.177:3300/0 conn(0x7fc3940b4cc0 msgr2=0x7fc39406f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 --2- 172.21.15.146:0/2191698694 >> v2:172.21.15.177:3300/0 conn(0x7fc3940b4cc0 0x7fc39406f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 >> v1:172.21.15.177:6789/0 conn(0x7fc3940b6030 legacy=0x7fc3940d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3940b50d0 con 0x7fc39406fc20 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fc38c0019f0 con 0x7fc39406fc20 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 >> v1:172.21.15.146:6789/0 conn(0x7fc39406fc20 legacy=0x7fc3940cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc380003b40 0x7fc380003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380004800 0x7fc380004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc380005260 0x7fc38000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc39408e020 con 0x7fc380005260 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc38000c1c0 con 0x7fc380004800 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc38000c3c0 con 0x7fc380003b40 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39abcd640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc380005260 0x7fc38000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc399bcb640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380004800 0x7fc380004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39a3cc640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc380003b40 0x7fc380003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39abcd640 1 -- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc380003b40 msgr2=0x7fc380003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39abcd640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc380003b40 0x7fc380003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39abcd640 1 -- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380004800 msgr2=0x7fc380004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39abcd640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380004800 0x7fc380004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.598+0000 7fc39abcd640 1 -- 172.21.15.146:0/2191698694 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc380003520 con 0x7fc380005260 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.599+0000 7fc39abcd640 1 -- 172.21.15.146:0/2191698694 reap_dead start 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.599+0000 7fc39abcd640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc380005260 0x7fc38000b9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc39406f240 tx=0x7fc380005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2b88c80b94926b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.599+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c00a450 con 0x7fc380005260 2024-04-25T15:20:43.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.599+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38c02f070 con 0x7fc380005260 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.599+0000 7fc3993ca640 1 -- 172.21.15.146:0/2191698694 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c02bc20 con 0x7fc380005260 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 -- 172.21.15.146:0/2191698694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc380005260 msgr2=0x7fc38000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc380005260 0x7fc38000b9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc39406f240 tx=0x7fc380005830 comp rx=0 tx=0).stop 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 -- 172.21.15.146:0/2191698694 shutdown_connections 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 --2- 172.21.15.146:0/2191698694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc380005260 0x7fc38000b9c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 -- 172.21.15.146:0/2191698694 >> 172.21.15.146:0/2191698694 conn(0x7fc3940b5430 msgr2=0x7fc3940b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 -- 172.21.15.146:0/2191698694 shutdown_connections 2024-04-25T15:20:43.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.600+0000 7fc39c630640 1 -- 172.21.15.146:0/2191698694 wait complete. 2024-04-25T15:20:43.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.602+0000 7fc39c630640 1 Processor -- start 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.602+0000 7fc39c630640 1 -- start start 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.603+0000 7fc39c630640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380005260 0x7fc39414c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.603+0000 7fc39c630640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc394162d40 0x7fc3941611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.603+0000 7fc39c630640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 0x7fc394158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.603+0000 7fc39c630640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc39406f860 con 0x7fc394158690 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.603+0000 7fc39c630640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc39406f6e0 con 0x7fc380005260 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.603+0000 7fc39c630640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc39406f9e0 con 0x7fc394162d40 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 0x7fc394158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 0x7fc394158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56002/0 (socket says 172.21.15.146:56002) 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 -- 172.21.15.146:0/527498860 learned_addr learned my addr 172.21.15.146:0/527498860 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc399bcb640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc394162d40 0x7fc3941611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39a3cc640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380005260 0x7fc39414c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 -- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc394162d40 msgr2=0x7fc3941611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc394162d40 0x7fc3941611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 -- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380005260 msgr2=0x7fc39414c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380005260 0x7fc39414c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 -- 172.21.15.146:0/527498860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc394158fe0 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc399bcb640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc394162d40 0x7fc3941611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39abcd640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 0x7fc394158aa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc38c002230 tx=0x7fc39408df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c045070 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38c02bbd0 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3941632e0 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c040480 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.604+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3941638a0 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.605+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39406a0e0 con 0x7fc394158690 2024-04-25T15:20:43.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.606+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc38c048070 con 0x7fc394158690 2024-04-25T15:20:43.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.606+0000 7fc38affd640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc3780744b0 0x7fc3780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.607+0000 7fc39a3cc640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc3780744b0 0x7fc3780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.607+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fc38c0b5660 con 0x7fc394158690 2024-04-25T15:20:43.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.607+0000 7fc39a3cc640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc3780744b0 0x7fc3780769c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc390000c00 tx=0x7fc390009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.611+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc394158fe0 con 0x7fc394158690 2024-04-25T15:20:43.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.785+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc39414cea0 con 0x7fc394158690 2024-04-25T15:20:43.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.785+0000 7fc38affd640 1 -- 172.21.15.146:0/527498860 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fc38c07fa90 con 0x7fc394158690 2024-04-25T15:20:43.787 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:43.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.787+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc3780744b0 msgr2=0x7fc3780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.787+0000 7fc39c630640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc3780744b0 0x7fc3780769c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc390000c00 tx=0x7fc390009040 comp rx=0 tx=0).stop 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.787+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 msgr2=0x7fc394158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.787+0000 7fc39c630640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 0x7fc394158aa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc38c002230 tx=0x7fc39408df10 comp rx=0 tx=0).stop 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 shutdown_connections 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc3780744b0 0x7fc3780769c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc394158690 0x7fc394158aa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc394162d40 0x7fc3941611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 --2- 172.21.15.146:0/527498860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc380005260 0x7fc39414c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 >> 172.21.15.146:0/527498860 conn(0x7fc3940b5430 msgr2=0x7fc3940b8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 shutdown_connections 2024-04-25T15:20:43.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.788+0000 7fc39c630640 1 -- 172.21.15.146:0/527498860 wait complete. 2024-04-25T15:20:43.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:43.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.910+0000 7f555ba25640 1 Processor -- start 2024-04-25T15:20:43.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.910+0000 7f555ba25640 1 -- start start 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f555ba25640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f555409bd80 0x7f55540ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f555ba25640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55540d2680 0x7f55540d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f555ba25640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55540d2fd0 con 0x7f555415e070 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f555ba25640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f55540ce920 con 0x7f55540d2680 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f555ba25640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f55540cec00 con 0x7f555409bd80 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55540d2680 0x7f55540d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55540d2680 0x7f55540d2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56008/0 (socket says 172.21.15.146:56008) 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 -- 172.21.15.146:0/171174711 learned_addr learned my addr 172.21.15.146:0/171174711 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5558fc0640 1 --2- 172.21.15.146:0/171174711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f555409bd80 0x7f55540ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 -- 172.21.15.146:0/171174711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f555409bd80 msgr2=0x7f55540ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 --2- 172.21.15.146:0/171174711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f555409bd80 0x7f55540ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 -- 172.21.15.146:0/171174711 >> v1:172.21.15.177:6789/0 conn(0x7f555415e070 legacy=0x7f55540cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:43.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.912+0000 7f5559fc2640 1 -- 172.21.15.146:0/171174711 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55540ceee0 con 0x7f55540d2680 2024-04-25T15:20:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.913+0000 7f5559fc2640 1 --2- 172.21.15.146:0/171174711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55540d2680 0x7f55540d2a90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f554c009870 tx=0x7f554c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331997f65e29c878 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.913+0000 7f554bfff640 1 -- 172.21.15.146:0/171174711 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c037070 con 0x7f55540d2680 2024-04-25T15:20:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.913+0000 7f554bfff640 1 -- 172.21.15.146:0/171174711 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f554c032440 con 0x7f55540d2680 2024-04-25T15:20:43.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.913+0000 7f554bfff640 1 -- 172.21.15.146:0/171174711 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c032810 con 0x7f55540d2680 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 -- 172.21.15.146:0/171174711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55540d2680 msgr2=0x7f55540d2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 --2- 172.21.15.146:0/171174711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55540d2680 0x7f55540d2a90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f554c009870 tx=0x7f554c02a210 comp rx=0 tx=0).stop 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 -- 172.21.15.146:0/171174711 shutdown_connections 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 --2- 172.21.15.146:0/171174711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55540d2680 0x7f55540d2a90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 --2- 172.21.15.146:0/171174711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f555409bd80 0x7f55540ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 -- 172.21.15.146:0/171174711 >> 172.21.15.146:0/171174711 conn(0x7f55540b7400 msgr2=0x7f55540b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 -- 172.21.15.146:0/171174711 shutdown_connections 2024-04-25T15:20:43.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.914+0000 7f555ba25640 1 -- 172.21.15.146:0/171174711 wait complete. 2024-04-25T15:20:43.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.916+0000 7f555ba25640 1 Processor -- start 2024-04-25T15:20:43.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.916+0000 7f555ba25640 1 -- start start 2024-04-25T15:20:43.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f555ba25640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 0x7f55540c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f555ba25640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55540d2680 0x7f55540cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f555ba25640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555415e070 0x7f55540b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f555ba25640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f55540ce5a0 con 0x7f555409bd80 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f555ba25640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55540cf4a0 con 0x7f555415e070 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f555ba25640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f555406cde0 con 0x7f55540d2680 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 0x7f55540c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 0x7f55540c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56024/0 (socket says 172.21.15.146:56024) 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 -- 172.21.15.146:0/3938329340 learned_addr learned my addr 172.21.15.146:0/3938329340 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f5558fc0640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55540d2680 0x7f55540cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f5559fc2640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555415e070 0x7f55540b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 -- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55540d2680 msgr2=0x7f55540cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55540d2680 0x7f55540cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 -- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555415e070 msgr2=0x7f55540b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555415e070 0x7f55540b1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f55597c1640 1 -- 172.21.15.146:0/3938329340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55540b3080 con 0x7f555409bd80 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.917+0000 7f5558fc0640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55540d2680 0x7f55540cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.918+0000 7f55597c1640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 0x7f55540c9c60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f555000ba80 tx=0x7f555000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.918+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555000cc00 con 0x7f555409bd80 2024-04-25T15:20:43.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.918+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555000cda0 con 0x7f555409bd80 2024-04-25T15:20:43.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.918+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55540b3360 con 0x7f555409bd80 2024-04-25T15:20:43.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.918+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550003c00 con 0x7f555409bd80 2024-04-25T15:20:43.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.918+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55540b0bf0 con 0x7f555409bd80 2024-04-25T15:20:43.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.919+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55540ce060 con 0x7f555409bd80 2024-04-25T15:20:43.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.919+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f555001b020 con 0x7f555409bd80 2024-04-25T15:20:43.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.920+0000 7f5549ffb640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f55380744b0 0x7f55380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:43.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.920+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5550095680 con 0x7f555409bd80 2024-04-25T15:20:43.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.920+0000 7f5558fc0640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f55380744b0 0x7f55380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:43.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.920+0000 7f5558fc0640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f55380744b0 0x7f55380769c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f55540b3c90 tx=0x7f5544008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:43.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:43.924+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55540ce060 con 0x7f555409bd80 2024-04-25T15:20:44.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.101+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f555406a0e0 con 0x7f555409bd80 2024-04-25T15:20:44.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.102+0000 7f5549ffb640 1 -- 172.21.15.146:0/3938329340 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f555005fab0 con 0x7f555409bd80 2024-04-25T15:20:44.104 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f55380744b0 msgr2=0x7f55380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f55380744b0 0x7f55380769c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f55540b3c90 tx=0x7f5544008040 comp rx=0 tx=0).stop 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 msgr2=0x7f55540c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 0x7f55540c9c60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f555000ba80 tx=0x7f555000bf40 comp rx=0 tx=0).stop 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 shutdown_connections 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f55380744b0 0x7f55380769c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555415e070 0x7f55540b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55540d2680 0x7f55540cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 --2- 172.21.15.146:0/3938329340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f555409bd80 0x7f55540c9c60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 >> 172.21.15.146:0/3938329340 conn(0x7f55540b7400 msgr2=0x7f555415a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:44.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 shutdown_connections 2024-04-25T15:20:44.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.104+0000 7f555ba25640 1 -- 172.21.15.146:0/3938329340 wait complete. 2024-04-25T15:20:44.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d3566e640 1 Processor -- start 2024-04-25T15:20:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d3566e640 1 -- start start 2024-04-25T15:20:44.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d3566e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d300bb450 0x7f5d300d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d3566e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d300b5230 con 0x7f5d30163c00 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d3566e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d3006c8c0 con 0x7f5d300d0de0 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d3566e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3006cba0 con 0x7f5d300bb450 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d2e7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d300bb450 0x7f5d300d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d2e7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d300bb450 0x7f5d300d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56034/0 (socket says 172.21.15.146:56034) 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.227+0000 7f5d2e7fc640 1 -- 172.21.15.146:0/4131335348 learned_addr learned my addr 172.21.15.146:0/4131335348 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.228+0000 7f5d2e7fc640 1 -- 172.21.15.146:0/4131335348 >> v1:172.21.15.177:6789/0 conn(0x7f5d300d0de0 legacy=0x7f5d300b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.228+0000 7f5d2e7fc640 1 -- 172.21.15.146:0/4131335348 >> v1:172.21.15.146:6789/0 conn(0x7f5d30163c00 legacy=0x7f5d3006d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.228+0000 7f5d2e7fc640 1 -- 172.21.15.146:0/4131335348 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3006ce80 con 0x7f5d300bb450 2024-04-25T15:20:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.228+0000 7f5d2e7fc640 1 --2- 172.21.15.146:0/4131335348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d300bb450 0x7f5d300d08a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5d180098a0 tx=0x7f5d1802a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d998d87220ba17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d2dffb640 1 -- 172.21.15.146:0/4131335348 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18037070 con 0x7f5d300bb450 2024-04-25T15:20:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d2dffb640 1 -- 172.21.15.146:0/4131335348 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d18032440 con 0x7f5d300bb450 2024-04-25T15:20:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d2dffb640 1 -- 172.21.15.146:0/4131335348 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18032810 con 0x7f5d300bb450 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 -- 172.21.15.146:0/4131335348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d300bb450 msgr2=0x7f5d300d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 --2- 172.21.15.146:0/4131335348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d300bb450 0x7f5d300d08a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5d180098a0 tx=0x7f5d1802a290 comp rx=0 tx=0).stop 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 -- 172.21.15.146:0/4131335348 shutdown_connections 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 --2- 172.21.15.146:0/4131335348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d300bb450 0x7f5d300d08a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 -- 172.21.15.146:0/4131335348 >> 172.21.15.146:0/4131335348 conn(0x7f5d300b7970 msgr2=0x7f5d300b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 -- 172.21.15.146:0/4131335348 shutdown_connections 2024-04-25T15:20:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.229+0000 7f5d3566e640 1 -- 172.21.15.146:0/4131335348 wait complete. 2024-04-25T15:20:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.231+0000 7f5d3566e640 1 Processor -- start 2024-04-25T15:20:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.231+0000 7f5d3566e640 1 -- start start 2024-04-25T15:20:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d300bb450 0x7f5d300c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d300d0de0 0x7f5d301f9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 0x7f5d3015b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5d300bb860 con 0x7f5d30163c00 2024-04-25T15:20:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3006c550 con 0x7f5d300d0de0 2024-04-25T15:20:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5d300bb9e0 con 0x7f5d300bb450 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 0x7f5d3015b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 0x7f5d3015b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56042/0 (socket says 172.21.15.146:56042) 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 -- 172.21.15.146:0/3648249231 learned_addr learned my addr 172.21.15.146:0/3648249231 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2effd640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d300bb450 0x7f5d300c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2e7fc640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d300d0de0 0x7f5d301f9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 -- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d300bb450 msgr2=0x7f5d300c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d300bb450 0x7f5d300c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 -- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d300d0de0 msgr2=0x7f5d301f9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d300d0de0 0x7f5d301f9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 -- 172.21.15.146:0/3648249231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d301f84f0 con 0x7f5d30163c00 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d2f7fe640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 0x7f5d3015b400 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000c680 tx=0x7f5d2000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20003b70 con 0x7f5d30163c00 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20003d10 con 0x7f5d30163c00 2024-04-25T15:20:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d301f87d0 con 0x7f5d30163c00 2024-04-25T15:20:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20005310 con 0x7f5d30163c00 2024-04-25T15:20:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d301f8d90 con 0x7f5d30163c00 2024-04-25T15:20:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.234+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3006a0e0 con 0x7f5d30163c00 2024-04-25T15:20:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.235+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d200054b0 con 0x7f5d30163c00 2024-04-25T15:20:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.236+0000 7f5d277fe640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5d100744b0 0x7f5d100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.236+0000 7f5d2effd640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5d100744b0 0x7f5d100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.236+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5d20095410 con 0x7f5d30163c00 2024-04-25T15:20:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.236+0000 7f5d2effd640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5d100744b0 0x7f5d100769c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d28000f60 tx=0x7f5d28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.241+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d2005f840 con 0x7f5d30163c00 2024-04-25T15:20:44.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.415+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5d300d3ed0 con 0x7f5d30163c00 2024-04-25T15:20:44.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.415+0000 7f5d277fe640 1 -- 172.21.15.146:0/3648249231 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f5d2005f140 con 0x7f5d30163c00 2024-04-25T15:20:44.417 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:44.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5d100744b0 msgr2=0x7f5d100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5d100744b0 0x7f5d100769c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d28000f60 tx=0x7f5d28009040 comp rx=0 tx=0).stop 2024-04-25T15:20:44.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 msgr2=0x7f5d3015b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 0x7f5d3015b400 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000c680 tx=0x7f5d2000cb40 comp rx=0 tx=0).stop 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 shutdown_connections 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5d100744b0 0x7f5d100769c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d30163c00 0x7f5d3015b400 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d300d0de0 0x7f5d301f9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 --2- 172.21.15.146:0/3648249231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d300bb450 0x7f5d300c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 >> 172.21.15.146:0/3648249231 conn(0x7f5d300b7970 msgr2=0x7f5d300d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 shutdown_connections 2024-04-25T15:20:44.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.418+0000 7f5d3566e640 1 -- 172.21.15.146:0/3648249231 wait complete. 2024-04-25T15:20:44.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:44.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.541+0000 7fecf8d9e640 1 Processor -- start 2024-04-25T15:20:44.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.541+0000 7fecf8d9e640 1 -- start start 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.541+0000 7fecf8d9e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40b2cd0 0x7fecf40cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf8d9e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf40cf7d0 con 0x7fecf40d2440 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf8d9e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf40cfab0 con 0x7fecf40d1920 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf8d9e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fecf408da90 con 0x7fecf40b2cd0 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf2d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fecf40d2440 0x7fecf4071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34694/0 (socket says 172.21.15.146:34694) 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf2d9d640 1 -- 172.21.15.146:0/2891368179 learned_addr learned my addr 172.21.15.146:0/2891368179 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:44.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf259c640 1 --2- 172.21.15.146:0/2891368179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40b2cd0 0x7fecf40cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109462069 0 0) 0x7fecf40cf7d0 con 0x7fecf40d2440 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecdc003520 con 0x7fecf40d2440 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491818096 0 0) 0x7fecf40cfab0 con 0x7fecf40d1920 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf40cf7d0 con 0x7fecf40d1920 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf259c640 1 -- 172.21.15.146:0/2891368179 >> v1:172.21.15.177:6789/0 conn(0x7fecf40d1920 legacy=0x7fecf40d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf259c640 1 -- 172.21.15.146:0/2891368179 >> v1:172.21.15.146:6789/0 conn(0x7fecf40d2440 legacy=0x7fecf4071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.542+0000 7fecf259c640 1 -- 172.21.15.146:0/2891368179 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf40cfd90 con 0x7fecf40b2cd0 2024-04-25T15:20:44.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.543+0000 7fecf259c640 1 --2- 172.21.15.146:0/2891368179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40b2cd0 0x7fecf40cf0f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fece40097c0 tx=0x7fece402a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3999bc465200e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.543+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4037070 con 0x7fecf40b2cd0 2024-04-25T15:20:44.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.543+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece4032440 con 0x7fecf40b2cd0 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.543+0000 7fecf159a640 1 -- 172.21.15.146:0/2891368179 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40327f0 con 0x7fecf40b2cd0 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2891368179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40b2cd0 msgr2=0x7fecf40cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2891368179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40b2cd0 0x7fecf40cf0f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fece40097c0 tx=0x7fece402a160 comp rx=0 tx=0).stop 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2891368179 shutdown_connections 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2891368179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40b2cd0 0x7fecf40cf0f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2891368179 >> 172.21.15.146:0/2891368179 conn(0x7fecf40b52e0 msgr2=0x7fecf40b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2891368179 shutdown_connections 2024-04-25T15:20:44.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.544+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2891368179 wait complete. 2024-04-25T15:20:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.546+0000 7fecf8d9e640 1 Processor -- start 2024-04-25T15:20:44.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.546+0000 7fecf8d9e640 1 -- start start 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf8d9e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecf40b2cd0 0x7fecf415c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf8d9e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40d1920 0x7fecf4158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf8d9e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 0x7fecf415a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf8d9e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4072790 con 0x7fecf40d2440 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf8d9e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4072610 con 0x7fecf40b2cd0 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf8d9e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fecf4072910 con 0x7fecf40d1920 2024-04-25T15:20:44.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf2d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 0x7fecf415a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf2d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 0x7fecf415a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56044/0 (socket says 172.21.15.146:56044) 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf2d9d640 1 -- 172.21.15.146:0/2552336405 learned_addr learned my addr 172.21.15.146:0/2552336405 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf1d9b640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40d1920 0x7fecf4158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.547+0000 7fecf259c640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecf40b2cd0 0x7fecf415c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf2d9d640 1 -- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40d1920 msgr2=0x7fecf4158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf2d9d640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40d1920 0x7fecf4158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf2d9d640 1 -- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecf40b2cd0 msgr2=0x7fecf415c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf2d9d640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecf40b2cd0 0x7fecf415c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf2d9d640 1 -- 172.21.15.146:0/2552336405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf415fcb0 con 0x7fecf40d2440 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf2d9d640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 0x7fecf415a010 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fecec00caf0 tx=0x7fecec00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec017070 con 0x7fecf40d2440 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecec00ac40 con 0x7fecf40d2440 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf415ff90 con 0x7fecf40d2440 2024-04-25T15:20:44.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec012400 con 0x7fecf40d2440 2024-04-25T15:20:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.548+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf4160550 con 0x7fecf40d2440 2024-04-25T15:20:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.549+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecf406a0e0 con 0x7fecf40d2440 2024-04-25T15:20:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.549+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fecec01a070 con 0x7fecf40d2440 2024-04-25T15:20:44.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.550+0000 7fece2ffd640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecd40744b0 0x7fecd40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.550+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fecec0952c0 con 0x7fecf40d2440 2024-04-25T15:20:44.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.550+0000 7fecf259c640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecd40744b0 0x7fecd40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.550+0000 7fecf259c640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecd40744b0 0x7fecd40769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fece4034000 tx=0x7fece402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.555+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fecec05f0a0 con 0x7fecf40d2440 2024-04-25T15:20:44.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.731+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fecf408d580 con 0x7fecf40d2440 2024-04-25T15:20:44.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.731+0000 7fece2ffd640 1 -- 172.21.15.146:0/2552336405 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fecec0633a0 con 0x7fecf40d2440 2024-04-25T15:20:44.734 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:44.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecd40744b0 msgr2=0x7fecd40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecd40744b0 0x7fecd40769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fece4034000 tx=0x7fece402e000 comp rx=0 tx=0).stop 2024-04-25T15:20:44.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 msgr2=0x7fecf415a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 0x7fecf415a010 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fecec00caf0 tx=0x7fecec00a060 comp rx=0 tx=0).stop 2024-04-25T15:20:44.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 shutdown_connections 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecd40744b0 0x7fecd40769c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecf40d2440 0x7fecf415a010 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecf40d1920 0x7fecf4158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 --2- 172.21.15.146:0/2552336405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecf40b2cd0 0x7fecf415c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.733+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 >> 172.21.15.146:0/2552336405 conn(0x7fecf40b52e0 msgr2=0x7fecf40d1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.734+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 shutdown_connections 2024-04-25T15:20:44.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.734+0000 7fecf8d9e640 1 -- 172.21.15.146:0/2552336405 wait complete. 2024-04-25T15:20:44.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:44.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.854+0000 7f7809fce640 1 Processor -- start 2024-04-25T15:20:44.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.854+0000 7f7809fce640 1 -- start start 2024-04-25T15:20:44.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f7809fce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78040cfbe0 0x7f78040cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f7809fce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f7809fce640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78040b4a60 con 0x7f78040b3920 2024-04-25T15:20:44.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f7809fce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f78040b5020 con 0x7f78040cfbe0 2024-04-25T15:20:44.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f7809fce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f78040b4d40 con 0x7f78040d19d0 2024-04-25T15:20:44.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f78037fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78040cfbe0 0x7f78040cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f78037fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78040cfbe0 0x7f78040cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56052/0 (socket says 172.21.15.146:56052) 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.855+0000 7f78037fe640 1 -- 172.21.15.146:0/482250078 learned_addr learned my addr 172.21.15.146:0/482250078 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f7802ffd640 1 --2- 172.21.15.146:0/482250078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f78037fe640 1 -- 172.21.15.146:0/482250078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 msgr2=0x7f78040d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f78037fe640 1 --2- 172.21.15.146:0/482250078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f78037fe640 1 -- 172.21.15.146:0/482250078 >> v1:172.21.15.146:6789/0 conn(0x7f78040b3920 legacy=0x7f78040b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f78037fe640 1 -- 172.21.15.146:0/482250078 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78040cead0 con 0x7f78040cfbe0 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f78037fe640 1 --2- 172.21.15.146:0/482250078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040cfbe0 0x7f78040cf260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009850 tx=0x7f77f802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc830ce90d7fc4e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.856+0000 7f78027fc640 1 -- 172.21.15.146:0/482250078 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8004230 con 0x7f78040cfbe0 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f78027fc640 1 -- 172.21.15.146:0/482250078 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f80043d0 con 0x7f78040cfbe0 2024-04-25T15:20:44.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f78027fc640 1 -- 172.21.15.146:0/482250078 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8003810 con 0x7f78040cfbe0 2024-04-25T15:20:44.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 -- 172.21.15.146:0/482250078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040cfbe0 msgr2=0x7f78040cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 --2- 172.21.15.146:0/482250078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040cfbe0 0x7f78040cf260 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009850 tx=0x7f77f802a2c0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 -- 172.21.15.146:0/482250078 shutdown_connections 2024-04-25T15:20:44.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 --2- 172.21.15.146:0/482250078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 --2- 172.21.15.146:0/482250078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040cfbe0 0x7f78040cf260 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 -- 172.21.15.146:0/482250078 >> 172.21.15.146:0/482250078 conn(0x7f78040b73b0 msgr2=0x7f78040b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:44.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 -- 172.21.15.146:0/482250078 shutdown_connections 2024-04-25T15:20:44.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.857+0000 7f7809fce640 1 -- 172.21.15.146:0/482250078 wait complete. 2024-04-25T15:20:44.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.859+0000 7f7809fce640 1 Processor -- start 2024-04-25T15:20:44.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.859+0000 7f7809fce640 1 -- start start 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.860+0000 7f7809fce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 0x7f780415ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.860+0000 7f7809fce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78040cfbe0 0x7f780409fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.860+0000 7f7809fce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.860+0000 7f7809fce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f78040cf7a0 con 0x7f78040b3920 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.860+0000 7f7809fce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78040d2320 con 0x7f78040cfbe0 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.860+0000 7f7809fce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f780409c6d0 con 0x7f78040d19d0 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 0x7f780415ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 0x7f780415ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56066/0 (socket says 172.21.15.146:56066) 2024-04-25T15:20:44.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 -- 172.21.15.146:0/20866419 learned_addr learned my addr 172.21.15.146:0/20866419 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f7803fff640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f7802ffd640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78040cfbe0 0x7f780409fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 -- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 msgr2=0x7f78040a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 -- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78040cfbe0 msgr2=0x7f780409fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78040cfbe0 0x7f780409fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 -- 172.21.15.146:0/20866419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78041f6540 con 0x7f78040b3920 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f7802ffd640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78040cfbe0 0x7f780409fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f78037fe640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 0x7f780415ce00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009970 tx=0x7f77f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8041030 con 0x7f78040b3920 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f8003c50 con 0x7f78040b3920 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78041f6820 con 0x7f78040b3920 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8003e00 con 0x7f78040b3920 2024-04-25T15:20:44.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.861+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78041f6de0 con 0x7f78040b3920 2024-04-25T15:20:44.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.862+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78040d12d0 con 0x7f78040b3920 2024-04-25T15:20:44.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.863+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f77f8044070 con 0x7f78040b3920 2024-04-25T15:20:44.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.863+0000 7f77ebfff640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77f00744b0 0x7f77f00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:44.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.863+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f77f80b6090 con 0x7f78040b3920 2024-04-25T15:20:44.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.863+0000 7f7802ffd640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77f00744b0 0x7f77f00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:44.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.864+0000 7f7802ffd640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77f00744b0 0x7f77f00769c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f78040cf2d0 tx=0x7f77f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:44.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:44.869+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78040d12d0 con 0x7f78040b3920 2024-04-25T15:20:45.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.055+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f780406a0e0 con 0x7f78040b3920 2024-04-25T15:20:45.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.055+0000 7f77ebfff640 1 -- 172.21.15.146:0/20866419 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f77f80804c0 con 0x7f78040b3920 2024-04-25T15:20:45.058 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:45.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77f00744b0 msgr2=0x7f77f00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77f00744b0 0x7f77f00769c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f78040cf2d0 tx=0x7f77f4008040 comp rx=0 tx=0).stop 2024-04-25T15:20:45.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 msgr2=0x7f780415ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 0x7f780415ce00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009970 tx=0x7f77f80046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 shutdown_connections 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f77f00744b0 0x7f77f00769c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f78040d19d0 0x7f78040a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78040cfbe0 0x7f780409fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 --2- 172.21.15.146:0/20866419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78040b3920 0x7f780415ce00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.057+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 >> 172.21.15.146:0/20866419 conn(0x7f78040b73b0 msgr2=0x7f78040d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.058+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 shutdown_connections 2024-04-25T15:20:45.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.058+0000 7f7809fce640 1 -- 172.21.15.146:0/20866419 wait complete. 2024-04-25T15:20:45.071 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:20:45.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:45.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-25T15:20:45.100 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:20:45.100 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:20:45.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:45.100 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:20:45.166 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:20:45.166 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:45.182 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:20:45.182 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:45.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:45.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.384+0000 7f720fa49640 1 Processor -- start 2024-04-25T15:20:45.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.384+0000 7f720fa49640 1 -- start start 2024-04-25T15:20:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720fa49640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7208153ca0 0x7f720815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720fa49640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720fa49640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72080d1410 con 0x7f72080cbc90 2024-04-25T15:20:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720fa49640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f72080cfd10 con 0x7f7208153ca0 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720fa49640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f72080cf420 con 0x7f72080d0ac0 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720cfe4640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7208153ca0 0x7f720815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:48240/0 (socket says 172.21.15.146:48240) 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 -- 172.21.15.146:0/829669239 learned_addr learned my addr 172.21.15.146:0/829669239 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 -- 172.21.15.146:0/829669239 >> v2:172.21.15.177:3300/0 conn(0x7f7208153ca0 msgr2=0x7f720815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 --2- 172.21.15.146:0/829669239 >> v2:172.21.15.177:3300/0 conn(0x7f7208153ca0 0x7f720815dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 -- 172.21.15.146:0/829669239 >> v1:172.21.15.177:6789/0 conn(0x7f72080cbc90 legacy=0x7f72080d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.386+0000 7f720dfe6640 1 -- 172.21.15.146:0/829669239 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72080cc270 con 0x7f72080d0ac0 2024-04-25T15:20:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.387+0000 7f720cfe4640 1 --2- 172.21.15.146:0/829669239 >> v2:172.21.15.177:3300/0 conn(0x7f7208153ca0 0x7f720815dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.387+0000 7f720dfe6640 1 --2- 172.21.15.146:0/829669239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080d0ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7200009e50 tx=0x7f720002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e78cf2910ee21090 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.387+0000 7f71fffff640 1 -- 172.21.15.146:0/829669239 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200036070 con 0x7f72080d0ac0 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.387+0000 7f71fffff640 1 -- 172.21.15.146:0/829669239 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7200031600 con 0x7f72080d0ac0 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.387+0000 7f71fffff640 1 -- 172.21.15.146:0/829669239 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72000319d0 con 0x7f72080d0ac0 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 -- 172.21.15.146:0/829669239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 msgr2=0x7f72080d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 --2- 172.21.15.146:0/829669239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080d0ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7200009e50 tx=0x7f720002a710 comp rx=0 tx=0).stop 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 -- 172.21.15.146:0/829669239 shutdown_connections 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 --2- 172.21.15.146:0/829669239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080d0ed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 --2- 172.21.15.146:0/829669239 >> v2:172.21.15.177:3300/0 conn(0x7f7208153ca0 0x7f720815dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 -- 172.21.15.146:0/829669239 >> 172.21.15.146:0/829669239 conn(0x7f72080ce0c0 msgr2=0x7f72080ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 -- 172.21.15.146:0/829669239 shutdown_connections 2024-04-25T15:20:45.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.388+0000 7f720fa49640 1 -- 172.21.15.146:0/829669239 wait complete. 2024-04-25T15:20:45.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.390+0000 7f720fa49640 1 Processor -- start 2024-04-25T15:20:45.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.390+0000 7f720fa49640 1 -- start start 2024-04-25T15:20:45.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.391+0000 7f720fa49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72080cbc90 0x7f72081472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.391+0000 7f720fa49640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.391+0000 7f720fa49640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 0x7f72080b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.391+0000 7f720fa49640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f720815e1a0 con 0x7f7208153ca0 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.391+0000 7f720fa49640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72080cf9a0 con 0x7f72080cbc90 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.391+0000 7f720fa49640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f72081540b0 con 0x7f72080d0ac0 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 0x7f72080b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 0x7f72080b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56070/0 (socket says 172.21.15.146:56070) 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 -- 172.21.15.146:0/1640786128 learned_addr learned my addr 172.21.15.146:0/1640786128 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720d7e5640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72080cbc90 0x7f72081472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720cfe4640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 -- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 msgr2=0x7f72080b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 -- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72080cbc90 msgr2=0x7f72081472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72080cbc90 0x7f72081472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 -- 172.21.15.146:0/1640786128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72080b6ff0 con 0x7f7208153ca0 2024-04-25T15:20:45.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720dfe6640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 0x7f72080b1880 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7200009e50 tx=0x7f7200004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200036050 con 0x7f7208153ca0 2024-04-25T15:20:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7200042070 con 0x7f7208153ca0 2024-04-25T15:20:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72081553b0 con 0x7f7208153ca0 2024-04-25T15:20:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200009510 con 0x7f7208153ca0 2024-04-25T15:20:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.392+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7208156f10 con 0x7f7208153ca0 2024-04-25T15:20:45.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.394+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72080b72d0 con 0x7f7208153ca0 2024-04-25T15:20:45.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.394+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7200007c50 con 0x7f7208153ca0 2024-04-25T15:20:45.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.394+0000 7f71fdffb640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f71ec0744b0 0x7f71ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.395+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f72000b6360 con 0x7f7208153ca0 2024-04-25T15:20:45.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.395+0000 7f720d7e5640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f71ec0744b0 0x7f71ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.395+0000 7f720d7e5640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f71ec0744b0 0x7f71ec0769c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f720806c600 tx=0x7f7204009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:45.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.400+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72000800b0 con 0x7f7208153ca0 2024-04-25T15:20:45.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.573+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f720806a0e0 con 0x7f7208153ca0 2024-04-25T15:20:45.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.575+0000 7f71fdffb640 1 -- 172.21.15.146:0/1640786128 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f72000843b0 con 0x7f7208153ca0 2024-04-25T15:20:45.576 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:45.576 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f71ec0744b0 msgr2=0x7f71ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f71ec0744b0 0x7f71ec0769c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f720806c600 tx=0x7f7204009040 comp rx=0 tx=0).stop 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 msgr2=0x7f72080b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 0x7f72080b1880 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7200009e50 tx=0x7f7200004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 shutdown_connections 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f71ec0744b0 0x7f71ec0769c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7208153ca0 0x7f72080b1880 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72080d0ac0 0x7f72080b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 --2- 172.21.15.146:0/1640786128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72080cbc90 0x7f72081472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 >> 172.21.15.146:0/1640786128 conn(0x7f72080ce0c0 msgr2=0x7f72080cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:45.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 shutdown_connections 2024-04-25T15:20:45.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.577+0000 7f720fa49640 1 -- 172.21.15.146:0/1640786128 wait complete. 2024-04-25T15:20:45.589 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:45.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:45.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.700+0000 7fa75156c640 1 Processor -- start 2024-04-25T15:20:45.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.700+0000 7fa75156c640 1 -- start start 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa75156c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa74c070500 0x7fa74c0b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa75156c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa74c0d10b0 0x7fa74c15cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa75156c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c06d560 0x7fa74c06d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa75156c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa74c0b0850 con 0x7fa74c070500 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa75156c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa74c06deb0 con 0x7fa74c0d10b0 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa75156c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c0b0520 con 0x7fa74c06d560 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.701+0000 7fa74affd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa74c070500 0x7fa74c0b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa74c070500 0x7fa74c0b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56086/0 (socket says 172.21.15.146:56086) 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 -- 172.21.15.146:0/186960099 learned_addr learned my addr 172.21.15.146:0/186960099 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74b7fe640 1 --2- 172.21.15.146:0/186960099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c06d560 0x7fa74c06d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa743fff640 1 --2- 172.21.15.146:0/186960099 >> v2:172.21.15.177:3300/0 conn(0x7fa74c0d10b0 0x7fa74c15cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 -- 172.21.15.146:0/186960099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c06d560 msgr2=0x7fa74c06d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 --2- 172.21.15.146:0/186960099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c06d560 0x7fa74c06d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 -- 172.21.15.146:0/186960099 >> v2:172.21.15.177:3300/0 conn(0x7fa74c0d10b0 msgr2=0x7fa74c15cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:45.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 --2- 172.21.15.146:0/186960099 >> v2:172.21.15.177:3300/0 conn(0x7fa74c0d10b0 0x7fa74c15cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.702+0000 7fa74affd640 1 -- 172.21.15.146:0/186960099 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c070d90 con 0x7fa74c070500 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa74affd640 1 --2- 172.21.15.146:0/186960099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c070500 0x7fa74c0b0ce0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0d1060 tx=0x7fa73c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c894da3826b51a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa74a7fc640 1 -- 172.21.15.146:0/186960099 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c037070 con 0x7fa74c070500 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa74a7fc640 1 -- 172.21.15.146:0/186960099 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73c032440 con 0x7fa74c070500 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa74a7fc640 1 -- 172.21.15.146:0/186960099 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c032810 con 0x7fa74c070500 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 -- 172.21.15.146:0/186960099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c070500 msgr2=0x7fa74c0b0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 --2- 172.21.15.146:0/186960099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c070500 0x7fa74c0b0ce0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0d1060 tx=0x7fa73c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 -- 172.21.15.146:0/186960099 shutdown_connections 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 --2- 172.21.15.146:0/186960099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c06d560 0x7fa74c06d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 --2- 172.21.15.146:0/186960099 >> v2:172.21.15.177:3300/0 conn(0x7fa74c0d10b0 0x7fa74c15cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 --2- 172.21.15.146:0/186960099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c070500 0x7fa74c0b0ce0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 -- 172.21.15.146:0/186960099 >> 172.21.15.146:0/186960099 conn(0x7fa74c0d1c10 msgr2=0x7fa74c0733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.703+0000 7fa75156c640 1 -- 172.21.15.146:0/186960099 shutdown_connections 2024-04-25T15:20:45.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.704+0000 7fa75156c640 1 -- 172.21.15.146:0/186960099 wait complete. 2024-04-25T15:20:45.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.705+0000 7fa75156c640 1 Processor -- start 2024-04-25T15:20:45.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.705+0000 7fa75156c640 1 -- start start 2024-04-25T15:20:45.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa75156c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 0x7fa74c154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa75156c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa74c070500 0x7fa74c0cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa75156c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c0d10b0 0x7fa74c0cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa75156c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c09f7d0 con 0x7fa74c06d560 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa75156c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c09f650 con 0x7fa74c070500 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa75156c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c0b4610 con 0x7fa74c0d10b0 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 0x7fa74c154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 0x7fa74c154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56102/0 (socket says 172.21.15.146:56102) 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 -- 172.21.15.146:0/2129862127 learned_addr learned my addr 172.21.15.146:0/2129862127 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74b7fe640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c0d10b0 0x7fa74c0cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa743fff640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa74c070500 0x7fa74c0cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 -- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c0d10b0 msgr2=0x7fa74c0cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c0d10b0 0x7fa74c0cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 -- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa74c070500 msgr2=0x7fa74c0cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa74c070500 0x7fa74c0cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 -- 172.21.15.146:0/2129862127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c15d960 con 0x7fa74c06d560 2024-04-25T15:20:45.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.707+0000 7fa74affd640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 0x7fa74c154920 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa73c000c00 tx=0x7fa73c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:45.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.709+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c041020 con 0x7fa74c06d560 2024-04-25T15:20:45.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.709+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73c0306c0 con 0x7fa74c06d560 2024-04-25T15:20:45.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.709+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74c151ce0 con 0x7fa74c06d560 2024-04-25T15:20:45.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.709+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c037040 con 0x7fa74c06d560 2024-04-25T15:20:45.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.709+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74c1522a0 con 0x7fa74c06d560 2024-04-25T15:20:45.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.710+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa74c06a0e0 con 0x7fa74c06d560 2024-04-25T15:20:45.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.710+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa73c044020 con 0x7fa74c06d560 2024-04-25T15:20:45.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.711+0000 7fa7437fe640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa72c0744b0 0x7fa72c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:45.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.711+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fa73c0b5ca0 con 0x7fa74c06d560 2024-04-25T15:20:45.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.711+0000 7fa743fff640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa72c0744b0 0x7fa72c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:45.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.711+0000 7fa743fff640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa72c0744b0 0x7fa72c0769c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa74c154630 tx=0x7fa734008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:45.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.716+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa73c04b030 con 0x7fa74c06d560 2024-04-25T15:20:45.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.893+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa74c14d620 con 0x7fa74c06d560 2024-04-25T15:20:45.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.893+0000 7fa7437fe640 1 -- 172.21.15.146:0/2129862127 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fa73c0880c0 con 0x7fa74c06d560 2024-04-25T15:20:45.894 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa72c0744b0 msgr2=0x7fa72c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa72c0744b0 0x7fa72c0769c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa74c154630 tx=0x7fa734008040 comp rx=0 tx=0).stop 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 msgr2=0x7fa74c154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 0x7fa74c154920 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa73c000c00 tx=0x7fa73c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 shutdown_connections 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa72c0744b0 0x7fa72c0769c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa74c0d10b0 0x7fa74c0cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa74c070500 0x7fa74c0cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 --2- 172.21.15.146:0/2129862127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa74c06d560 0x7fa74c154920 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:45.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 >> 172.21.15.146:0/2129862127 conn(0x7fa74c0d1c10 msgr2=0x7fa74c1605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:45.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 shutdown_connections 2024-04-25T15:20:45.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:45.895+0000 7fa75156c640 1 -- 172.21.15.146:0/2129862127 wait complete. 2024-04-25T15:20:45.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:45.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:20:45.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:20:46.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.022+0000 7fbe496e2640 1 Processor -- start 2024-04-25T15:20:46.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.022+0000 7fbe496e2640 1 -- start start 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe496e2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe44163c00 0x7fbe4406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe496e2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbe440d0fb0 0x7fbe440b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe496e2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe4406dea0 con 0x7fbe440bb450 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe496e2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbe440b5230 con 0x7fbe44163c00 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe496e2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbe4406c8c0 con 0x7fbe440d0fb0 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe44163c00 0x7fbe4406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe44163c00 0x7fbe4406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56112/0 (socket says 172.21.15.146:56112) 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 -- 172.21.15.146:0/892547709 learned_addr learned my addr 172.21.15.146:0/892547709 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe437fe640 1 --2- 172.21.15.146:0/892547709 >> v2:172.21.15.177:3300/0 conn(0x7fbe440d0fb0 0x7fbe440b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 -- 172.21.15.146:0/892547709 >> v2:172.21.15.177:3300/0 conn(0x7fbe440d0fb0 msgr2=0x7fbe440b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 --2- 172.21.15.146:0/892547709 >> v2:172.21.15.177:3300/0 conn(0x7fbe440d0fb0 0x7fbe440b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 -- 172.21.15.146:0/892547709 >> v1:172.21.15.177:6789/0 conn(0x7fbe440bb450 legacy=0x7fbe440d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.024+0000 7fbe42ffd640 1 -- 172.21.15.146:0/892547709 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4406cba0 con 0x7fbe44163c00 2024-04-25T15:20:46.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.025+0000 7fbe42ffd640 1 --2- 172.21.15.146:0/892547709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe44163c00 0x7fbe4406d960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbe440d22c0 tx=0x7fbe3402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826e24fe90c73e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.025+0000 7fbe41ffb640 1 -- 172.21.15.146:0/892547709 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34036070 con 0x7fbe44163c00 2024-04-25T15:20:46.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.025+0000 7fbe41ffb640 1 -- 172.21.15.146:0/892547709 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe34031600 con 0x7fbe44163c00 2024-04-25T15:20:46.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.025+0000 7fbe41ffb640 1 -- 172.21.15.146:0/892547709 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe340319d0 con 0x7fbe44163c00 2024-04-25T15:20:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 -- 172.21.15.146:0/892547709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe44163c00 msgr2=0x7fbe4406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 --2- 172.21.15.146:0/892547709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe44163c00 0x7fbe4406d960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbe440d22c0 tx=0x7fbe3402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 -- 172.21.15.146:0/892547709 shutdown_connections 2024-04-25T15:20:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 --2- 172.21.15.146:0/892547709 >> v2:172.21.15.177:3300/0 conn(0x7fbe440d0fb0 0x7fbe440b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 --2- 172.21.15.146:0/892547709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe44163c00 0x7fbe4406d960 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 -- 172.21.15.146:0/892547709 >> 172.21.15.146:0/892547709 conn(0x7fbe440b7970 msgr2=0x7fbe440b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 -- 172.21.15.146:0/892547709 shutdown_connections 2024-04-25T15:20:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.026+0000 7fbe496e2640 1 -- 172.21.15.146:0/892547709 wait complete. 2024-04-25T15:20:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.028+0000 7fbe496e2640 1 Processor -- start 2024-04-25T15:20:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.028+0000 7fbe496e2640 1 -- start start 2024-04-25T15:20:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe496e2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 0x7fbe44150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe496e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe440d0fb0 0x7fbe4414d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe496e2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe44163c00 0x7fbe44150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe496e2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4406c550 con 0x7fbe440bb450 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe496e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4406d160 con 0x7fbe440d0fb0 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe496e2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbe440d13c0 con 0x7fbe44163c00 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 0x7fbe44150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 0x7fbe44150bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56126/0 (socket says 172.21.15.146:56126) 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 -- 172.21.15.146:0/1454556264 learned_addr learned my addr 172.21.15.146:0/1454556264 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe427fc640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe440d0fb0 0x7fbe4414d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe437fe640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe44163c00 0x7fbe44150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 -- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe44163c00 msgr2=0x7fbe44150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe44163c00 0x7fbe44150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 -- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe440d0fb0 msgr2=0x7fbe4414d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe440d0fb0 0x7fbe4414d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.029+0000 7fbe42ffd640 1 -- 172.21.15.146:0/1454556264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe440ca970 con 0x7fbe440bb450 2024-04-25T15:20:46.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.030+0000 7fbe42ffd640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 0x7fbe44150bf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009f40 tx=0x7fbe34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.030+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3403f020 con 0x7fbe440bb450 2024-04-25T15:20:46.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.030+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe34037040 con 0x7fbe440bb450 2024-04-25T15:20:46.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.030+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4414e210 con 0x7fbe440bb450 2024-04-25T15:20:46.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.030+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34036070 con 0x7fbe440bb450 2024-04-25T15:20:46.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.030+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe4414e7d0 con 0x7fbe440bb450 2024-04-25T15:20:46.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.031+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe440cb670 con 0x7fbe440bb450 2024-04-25T15:20:46.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.031+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbe34042020 con 0x7fbe440bb450 2024-04-25T15:20:46.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.032+0000 7fbe2b7fe640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe300744e0 0x7fbe300769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.032+0000 7fbe427fc640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe300744e0 0x7fbe300769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.032+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fbe340b5a00 con 0x7fbe440bb450 2024-04-25T15:20:46.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.032+0000 7fbe427fc640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe300744e0 0x7fbe300769f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe440d5770 tx=0x7fbe38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.037+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe440cb670 con 0x7fbe440bb450 2024-04-25T15:20:46.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.210+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbe4406a0e0 con 0x7fbe440bb450 2024-04-25T15:20:46.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.210+0000 7fbe2b7fe640 1 -- 172.21.15.146:0/1454556264 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fbe34049030 con 0x7fbe440bb450 2024-04-25T15:20:46.211 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:46.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe300744e0 msgr2=0x7fbe300769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe300744e0 0x7fbe300769f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe440d5770 tx=0x7fbe38009040 comp rx=0 tx=0).stop 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 msgr2=0x7fbe44150bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 0x7fbe44150bf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009f40 tx=0x7fbe34004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 shutdown_connections 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe300744e0 0x7fbe300769f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe44163c00 0x7fbe44150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe440d0fb0 0x7fbe4414d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 --2- 172.21.15.146:0/1454556264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe440bb450 0x7fbe44150bf0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 >> 172.21.15.146:0/1454556264 conn(0x7fbe440b7970 msgr2=0x7fbe4415ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 shutdown_connections 2024-04-25T15:20:46.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.212+0000 7fbe496e2640 1 -- 172.21.15.146:0/1454556264 wait complete. 2024-04-25T15:20:46.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.375+0000 7f311da67640 1 Processor -- start 2024-04-25T15:20:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.375+0000 7f311da67640 1 -- start start 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f311da67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 0x7f311806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f311da67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f311806dea0 con 0x7f31180bb450 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f311da67640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31180b5230 con 0x7f31180d0fb0 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f311da67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f311806c8c0 con 0x7f3118163c00 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f3116ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f31180bb450 0x7f31180d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34710/0 (socket says 172.21.15.146:34710) 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f3116ffd640 1 -- 172.21.15.146:0/4229550977 learned_addr learned my addr 172.21.15.146:0/4229550977 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f31177fe640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 0x7f311806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186083669 0 0) 0x7f311806dea0 con 0x7f31180bb450 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.376+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3100003520 con 0x7f31180bb450 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2857072219 0 0) 0x7f3100003520 con 0x7f31180bb450 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 msgr2=0x7f311806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 0x7f311806d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 >> v1:172.21.15.177:6789/0 conn(0x7f31180d0fb0 legacy=0x7f31180b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311806cba0 con 0x7f31180bb450 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f310c001670 con 0x7f31180bb450 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 >> v1:172.21.15.146:6789/0 conn(0x7f31180bb450 legacy=0x7f31180d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 0x7f31000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3100003f60 0x7f3100004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f3100005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f311806c8c0 con 0x7f3100004d40 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f310000bc00 con 0x7f3100003f60 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f310000be00 con 0x7f3118163c00 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3117fff640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f3100005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f31177fe640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 0x7f31000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3116ffd640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3100003f60 0x7f3100004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3117fff640 1 -- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 msgr2=0x7f31000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3117fff640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118163c00 0x7f31000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3117fff640 1 -- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3100003f60 msgr2=0x7f3100004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3117fff640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3100003f60 0x7f3100004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.377+0000 7f3117fff640 1 -- 172.21.15.146:0/4229550977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3100003520 con 0x7f3100004d40 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.378+0000 7f3117fff640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f3100005150 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3110001e20 tx=0x7f3100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd28c655cb4aca34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.378+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110001a70 con 0x7f3100004d40 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.378+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311002ae40 con 0x7f3100004d40 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.378+0000 7f31167fc640 1 -- 172.21.15.146:0/4229550977 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110003bb0 con 0x7f3100004d40 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f311da67640 1 -- 172.21.15.146:0/4229550977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 msgr2=0x7f3100005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f311da67640 1 --2- 172.21.15.146:0/4229550977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f3100005150 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3110001e20 tx=0x7f3100005830 comp rx=0 tx=0).stop 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f3117fff640 1 -- 172.21.15.146:0/4229550977 reap_dead start 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f311da67640 1 -- 172.21.15.146:0/4229550977 shutdown_connections 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f311da67640 1 -- 172.21.15.146:0/4229550977 >> 172.21.15.146:0/4229550977 conn(0x7f31180b7970 msgr2=0x7f31180b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f311da67640 1 -- 172.21.15.146:0/4229550977 shutdown_connections 2024-04-25T15:20:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.379+0000 7f311da67640 1 -- 172.21.15.146:0/4229550977 wait complete. 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.381+0000 7f311da67640 1 Processor -- start 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.381+0000 7f311da67640 1 -- start start 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f311da67640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f31180cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f311da67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118156770 0x7f31181f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f311da67640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3118156b80 0x7f3118163350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f311da67640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f311806c550 con 0x7f3100004d40 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f311da67640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f311806d160 con 0x7f3118156b80 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f311da67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f31180d13c0 con 0x7f3118156770 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f31177fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f31180cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f31177fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f31180cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56144/0 (socket says 172.21.15.146:56144) 2024-04-25T15:20:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f31177fe640 1 -- 172.21.15.146:0/206615644 learned_addr learned my addr 172.21.15.146:0/206615644 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f3117fff640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3118156b80 0x7f3118163350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.382+0000 7f3116ffd640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118156770 0x7f31181f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f31177fe640 1 -- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118156770 msgr2=0x7f31181f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f31177fe640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118156770 0x7f31181f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f31177fe640 1 -- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3118156b80 msgr2=0x7f3118163350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f31177fe640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3118156b80 0x7f3118163350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f31177fe640 1 -- 172.21.15.146:0/206615644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31181622b0 con 0x7f3100004d40 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f3117fff640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3118156b80 0x7f3118163350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f3116ffd640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118156770 0x7f31181f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f31177fe640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f31180cfa10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3108011940 tx=0x7f3108011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310800faf0 con 0x7f3100004d40 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f310800fc90 con 0x7f3100004d40 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3118162590 con 0x7f3100004d40 2024-04-25T15:20:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108014440 con 0x7f3100004d40 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.383+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3118162b50 con 0x7f3100004d40 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.384+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311806a0e0 con 0x7f3100004d40 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.385+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3108012040 con 0x7f3100004d40 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.385+0000 7f30fffff640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31040744e0 0x7f31040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.385+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f3108095c60 con 0x7f3100004d40 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.385+0000 7f3116ffd640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31040744e0 0x7f31040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.386+0000 7f3116ffd640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31040744e0 0x7f31040769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f31180b8fe0 tx=0x7f310c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.390+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3108019030 con 0x7f3100004d40 2024-04-25T15:20:46.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.566+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f31180d3ed0 con 0x7f3100004d40 2024-04-25T15:20:46.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.567+0000 7f30fffff640 1 -- 172.21.15.146:0/206615644 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f3108060090 con 0x7f3100004d40 2024-04-25T15:20:46.567 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31040744e0 msgr2=0x7f31040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31040744e0 0x7f31040769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f31180b8fe0 tx=0x7f310c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 msgr2=0x7f31180cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f31180cfa10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3108011940 tx=0x7f3108011e00 comp rx=0 tx=0).stop 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 shutdown_connections 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f31040744e0 0x7f31040769f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3118156b80 0x7f3118163350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3118156770 0x7f31181f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 --2- 172.21.15.146:0/206615644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3100004d40 0x7f31180cfa10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 >> 172.21.15.146:0/206615644 conn(0x7f31180b7970 msgr2=0x7f3118164010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 shutdown_connections 2024-04-25T15:20:46.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.569+0000 7f311da67640 1 -- 172.21.15.146:0/206615644 wait complete. 2024-04-25T15:20:46.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:46.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.691+0000 7fd765284640 1 Processor -- start 2024-04-25T15:20:46.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.691+0000 7fd765284640 1 -- start start 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd765284640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7600d2680 0x7fd7600d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd765284640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7600d2fd0 con 0x7fd76009bd80 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd765284640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7600ce920 con 0x7fd76015e070 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd765284640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd7600cec00 con 0x7fd7600d2680 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7600d2680 0x7fd7600d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75e7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd76009bd80 0x7fd7600ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34726/0 (socket says 172.21.15.146:34726) 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7600d2680 0x7fd7600d2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56160/0 (socket says 172.21.15.146:56160) 2024-04-25T15:20:46.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75e7fc640 1 -- 172.21.15.146:0/3533048007 learned_addr learned my addr 172.21.15.146:0/3533048007 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75f7fe640 1 -- 172.21.15.146:0/3533048007 >> v1:172.21.15.177:6789/0 conn(0x7fd76015e070 legacy=0x7fd7600cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75f7fe640 1 -- 172.21.15.146:0/3533048007 >> v1:172.21.15.146:6789/0 conn(0x7fd76009bd80 legacy=0x7fd7600ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75f7fe640 1 -- 172.21.15.146:0/3533048007 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7600ceee0 con 0x7fd7600d2680 2024-04-25T15:20:46.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.693+0000 7fd75dffb640 1 -- 172.21.15.146:0/3533048007 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410029334 0 0) 0x7fd754001110 con 0x7fd76009bd80 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.694+0000 7fd75f7fe640 1 --2- 172.21.15.146:0/3533048007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7600d2a90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd758009d90 tx=0x7fd75802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae01ef174a0f0b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.694+0000 7fd75dffb640 1 -- 172.21.15.146:0/3533048007 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758007740 con 0x7fd7600d2680 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.694+0000 7fd75dffb640 1 -- 172.21.15.146:0/3533048007 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7580078e0 con 0x7fd7600d2680 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.694+0000 7fd75dffb640 1 -- 172.21.15.146:0/3533048007 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758007cb0 con 0x7fd7600d2680 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 -- 172.21.15.146:0/3533048007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 msgr2=0x7fd7600d2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 --2- 172.21.15.146:0/3533048007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7600d2a90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd758009d90 tx=0x7fd75802a6f0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 -- 172.21.15.146:0/3533048007 shutdown_connections 2024-04-25T15:20:46.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 --2- 172.21.15.146:0/3533048007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7600d2a90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 -- 172.21.15.146:0/3533048007 >> 172.21.15.146:0/3533048007 conn(0x7fd7600b7400 msgr2=0x7fd7600b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:46.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 -- 172.21.15.146:0/3533048007 shutdown_connections 2024-04-25T15:20:46.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.695+0000 7fd765284640 1 -- 172.21.15.146:0/3533048007 wait complete. 2024-04-25T15:20:46.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.697+0000 7fd765284640 1 Processor -- start 2024-04-25T15:20:46.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.697+0000 7fd765284640 1 -- start start 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd765284640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd76009bd80 0x7fd7601e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd765284640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7601538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd765284640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd76015e070 0x7fd760158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd765284640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd76006cde0 con 0x7fd7600d2680 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd765284640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd7600cf4a0 con 0x7fd76015e070 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd765284640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd76006cf60 con 0x7fd76009bd80 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd75e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7601538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd75e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7601538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56162/0 (socket says 172.21.15.146:56162) 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd75e7fc640 1 -- 172.21.15.146:0/2846865772 learned_addr learned my addr 172.21.15.146:0/2846865772 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd75effd640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd76009bd80 0x7fd7601e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.698+0000 7fd75f7fe640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd76015e070 0x7fd760158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd75e7fc640 1 -- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd76009bd80 msgr2=0x7fd7601e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd75e7fc640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd76009bd80 0x7fd7601e7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd75e7fc640 1 -- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd76015e070 msgr2=0x7fd760158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd75e7fc640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd76015e070 0x7fd760158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd75e7fc640 1 -- 172.21.15.146:0/2846865772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7601e8420 con 0x7fd7600d2680 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd75e7fc640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7601538e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd75400c420 tx=0x7fd75400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540074e0 con 0x7fd7600d2680 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7540047e0 con 0x7fd7600d2680 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7601e8700 con 0x7fd7600d2680 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754004c10 con 0x7fd7600d2680 2024-04-25T15:20:46.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.699+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7600c9820 con 0x7fd7600d2680 2024-04-25T15:20:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.700+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd76006a0e0 con 0x7fd7600d2680 2024-04-25T15:20:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.701+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd754011070 con 0x7fd7600d2680 2024-04-25T15:20:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.701+0000 7fd7477fe640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd74c0744b0 0x7fd74c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.701+0000 7fd75effd640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd74c0744b0 0x7fd74c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.701+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fd754095160 con 0x7fd7600d2680 2024-04-25T15:20:46.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.702+0000 7fd75effd640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd74c0744b0 0x7fd74c0769c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd750000fb0 tx=0x7fd750009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:46.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.706+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7600d2fd0 con 0x7fd7600d2680 2024-04-25T15:20:46.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.884+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd7601e8190 con 0x7fd7600d2680 2024-04-25T15:20:46.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.885+0000 7fd7477fe640 1 -- 172.21.15.146:0/2846865772 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fd75405f620 con 0x7fd7600d2680 2024-04-25T15:20:46.886 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:46.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd74c0744b0 msgr2=0x7fd74c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd74c0744b0 0x7fd74c0769c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd750000fb0 tx=0x7fd750009040 comp rx=0 tx=0).stop 2024-04-25T15:20:46.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 msgr2=0x7fd7601538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:46.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7601538e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd75400c420 tx=0x7fd75400c8e0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 shutdown_connections 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd74c0744b0 0x7fd74c0769c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd76015e070 0x7fd760158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7600d2680 0x7fd7601538e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 --2- 172.21.15.146:0/2846865772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd76009bd80 0x7fd7601e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 >> 172.21.15.146:0/2846865772 conn(0x7fd7600b7400 msgr2=0x7fd7600684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 shutdown_connections 2024-04-25T15:20:46.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:46.887+0000 7fd765284640 1 -- 172.21.15.146:0/2846865772 wait complete. 2024-04-25T15:20:46.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:47.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.008+0000 7f14e6f9d640 1 Processor -- start 2024-04-25T15:20:47.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.009+0000 7f14e6f9d640 1 -- start start 2024-04-25T15:20:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e6f9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14e015e8a0 0x7f14e015ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e6f9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f14e00af710 0x7f14e00afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e6f9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e006d560 0x7f14e006d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e6f9d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f14e00af2a0 con 0x7f14e00af710 2024-04-25T15:20:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e6f9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f14e006deb0 con 0x7f14e015e8a0 2024-04-25T15:20:47.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e6f9d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f14e00ae690 con 0x7f14e006d560 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f14e00af710 0x7f14e00afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f14e00af710 0x7f14e00afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56168/0 (socket says 172.21.15.146:56168) 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 -- 172.21.15.146:0/4167188601 learned_addr learned my addr 172.21.15.146:0/4167188601 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e5f9b640 1 --2- 172.21.15.146:0/4167188601 >> v2:172.21.15.177:3300/0 conn(0x7f14e015e8a0 0x7f14e015ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e679c640 1 --2- 172.21.15.146:0/4167188601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e006d560 0x7f14e006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 -- 172.21.15.146:0/4167188601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e006d560 msgr2=0x7f14e006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 --2- 172.21.15.146:0/4167188601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e006d560 0x7f14e006d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 -- 172.21.15.146:0/4167188601 >> v2:172.21.15.177:3300/0 conn(0x7f14e015e8a0 msgr2=0x7f14e015ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 --2- 172.21.15.146:0/4167188601 >> v2:172.21.15.177:3300/0 conn(0x7f14e015e8a0 0x7f14e015ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.010+0000 7f14e579a640 1 -- 172.21.15.146:0/4167188601 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e00ae970 con 0x7f14e00af710 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.011+0000 7f14e579a640 1 --2- 172.21.15.146:0/4167188601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e00af710 0x7f14e00afb20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f14e00d2640 tx=0x7f14d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c516618f709ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.011+0000 7f14e4f99640 1 -- 172.21.15.146:0/4167188601 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4037070 con 0x7f14e00af710 2024-04-25T15:20:47.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.011+0000 7f14e4f99640 1 -- 172.21.15.146:0/4167188601 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d4032440 con 0x7f14e00af710 2024-04-25T15:20:47.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.011+0000 7f14e4f99640 1 -- 172.21.15.146:0/4167188601 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4032810 con 0x7f14e00af710 2024-04-25T15:20:47.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 -- 172.21.15.146:0/4167188601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e00af710 msgr2=0x7f14e00afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/4167188601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e00af710 0x7f14e00afb20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f14e00d2640 tx=0x7f14d402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 -- 172.21.15.146:0/4167188601 shutdown_connections 2024-04-25T15:20:47.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/4167188601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e006d560 0x7f14e006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/4167188601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e00af710 0x7f14e00afb20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/4167188601 >> v2:172.21.15.177:3300/0 conn(0x7f14e015e8a0 0x7f14e015ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 -- 172.21.15.146:0/4167188601 >> 172.21.15.146:0/4167188601 conn(0x7f14e00732b0 msgr2=0x7f14e00736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 -- 172.21.15.146:0/4167188601 shutdown_connections 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.012+0000 7f14e6f9d640 1 -- 172.21.15.146:0/4167188601 wait complete. 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.014+0000 7f14e6f9d640 1 Processor -- start 2024-04-25T15:20:47.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.014+0000 7f14e6f9d640 1 -- start start 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e6f9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e006d560 0x7f14e013f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e6f9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e00af710 0x7f14e01f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e6f9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 0x7f14e01f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e6f9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f14e0161810 con 0x7f14e015e8a0 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e6f9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14e00aef30 con 0x7f14e006d560 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e6f9d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f14e0161990 con 0x7f14e00af710 2024-04-25T15:20:47.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e679c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 0x7f14e01f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e679c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 0x7f14e01f81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56176/0 (socket says 172.21.15.146:56176) 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.015+0000 7f14e679c640 1 -- 172.21.15.146:0/465655017 learned_addr learned my addr 172.21.15.146:0/465655017 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e579a640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e00af710 0x7f14e01f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e5f9b640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e006d560 0x7f14e013f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e679c640 1 -- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e00af710 msgr2=0x7f14e01f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e679c640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e00af710 0x7f14e01f7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e679c640 1 -- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e006d560 msgr2=0x7f14e013f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e679c640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e006d560 0x7f14e013f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e679c640 1 -- 172.21.15.146:0/465655017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e01428b0 con 0x7f14e015e8a0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e579a640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e00af710 0x7f14e01f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e679c640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 0x7f14e01f81b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f14d000d600 tx=0x7f14d000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d00125f0 con 0x7f14e015e8a0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d0019070 con 0x7f14e015e8a0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e0142b90 con 0x7f14e015e8a0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d00129f0 con 0x7f14e015e8a0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.016+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e0143150 con 0x7f14e015e8a0 2024-04-25T15:20:47.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.017+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e0050740 con 0x7f14e015e8a0 2024-04-25T15:20:47.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.018+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f14d0012b90 con 0x7f14e015e8a0 2024-04-25T15:20:47.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.018+0000 7f14de7fc640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f14cc0744e0 0x7f14cc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.018+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f14d00959f0 con 0x7f14e015e8a0 2024-04-25T15:20:47.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.018+0000 7f14e5f9b640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f14cc0744e0 0x7f14cc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.019+0000 7f14e5f9b640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f14cc0744e0 0x7f14cc0769f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009940 tx=0x7f14d8003cd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.023+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14d005fda0 con 0x7f14e015e8a0 2024-04-25T15:20:47.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.209+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f14e0142640 con 0x7f14e015e8a0 2024-04-25T15:20:47.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.210+0000 7f14de7fc640 1 -- 172.21.15.146:0/465655017 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f14d005f6a0 con 0x7f14e015e8a0 2024-04-25T15:20:47.211 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f14cc0744e0 msgr2=0x7f14cc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f14cc0744e0 0x7f14cc0769f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009940 tx=0x7f14d8003cd0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 msgr2=0x7f14e01f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 0x7f14e01f81b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f14d000d600 tx=0x7f14d000dac0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 shutdown_connections 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f14cc0744e0 0x7f14cc0769f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e015e8a0 0x7f14e01f81b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e00af710 0x7f14e01f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 --2- 172.21.15.146:0/465655017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e006d560 0x7f14e013f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 >> 172.21.15.146:0/465655017 conn(0x7f14e00732b0 msgr2=0x7f14e00b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:47.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 shutdown_connections 2024-04-25T15:20:47.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.212+0000 7f14e6f9d640 1 -- 172.21.15.146:0/465655017 wait complete. 2024-04-25T15:20:47.224 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:20:47.224 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:47.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:47.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.389+0000 7f430cbbe640 1 Processor -- start 2024-04-25T15:20:47.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.389+0000 7f430cbbe640 1 -- start start 2024-04-25T15:20:47.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f430cbbe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43080b4df0 0x7f4308161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f430cbbe640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4308162520 con 0x7f4308162890 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f430cbbe640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43080d1790 con 0x7f4308162110 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f430cbbe640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f43080d23a0 con 0x7f43080b4df0 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f430659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4308162890 0x7f430808d670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34740/0 (socket says 172.21.15.146:34740) 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f430659c640 1 -- 172.21.15.146:0/2514926698 learned_addr learned my addr 172.21.15.146:0/2514926698 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f42fffff640 1 --2- 172.21.15.146:0/2514926698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43080b4df0 0x7f4308161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221544194 0 0) 0x7f42f8001290 con 0x7f4308162890 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.390+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f4003520 con 0x7f4308162890 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1165563801 0 0) 0x7f43080d1790 con 0x7f4308162110 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f8001290 con 0x7f4308162110 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f42fffff640 1 -- 172.21.15.146:0/2514926698 >> v1:172.21.15.177:6789/0 conn(0x7f4308162110 legacy=0x7f43080ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f42fffff640 1 -- 172.21.15.146:0/2514926698 >> v1:172.21.15.146:6789/0 conn(0x7f4308162890 legacy=0x7f430808d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2115556848 0 0) 0x7f42f4003520 con 0x7f4308162890 2024-04-25T15:20:47.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f42fffff640 1 -- 172.21.15.146:0/2514926698 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43080d1a70 con 0x7f43080b4df0 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.391+0000 7f42fffff640 1 --2- 172.21.15.146:0/2514926698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43080b4df0 0x7f4308161bd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42f00097d0 tx=0x7f42f002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ccc9fb46918c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.392+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0037070 con 0x7f43080b4df0 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.392+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f002ab90 con 0x7f43080b4df0 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.392+0000 7f4305d9b640 1 -- 172.21.15.146:0/2514926698 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f80019c0 con 0x7f43080b4df0 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.392+0000 7f430cbbe640 1 -- 172.21.15.146:0/2514926698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43080b4df0 msgr2=0x7f4308161bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.392+0000 7f430cbbe640 1 --2- 172.21.15.146:0/2514926698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43080b4df0 0x7f4308161bd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42f00097d0 tx=0x7f42f002a100 comp rx=0 tx=0).stop 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.393+0000 7f430cbbe640 1 -- 172.21.15.146:0/2514926698 shutdown_connections 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.393+0000 7f430cbbe640 1 --2- 172.21.15.146:0/2514926698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f43080b4df0 0x7f4308161bd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.393+0000 7f430cbbe640 1 -- 172.21.15.146:0/2514926698 >> 172.21.15.146:0/2514926698 conn(0x7f43080b52e0 msgr2=0x7f43080b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.393+0000 7f430cbbe640 1 -- 172.21.15.146:0/2514926698 shutdown_connections 2024-04-25T15:20:47.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.393+0000 7f430cbbe640 1 -- 172.21.15.146:0/2514926698 wait complete. 2024-04-25T15:20:47.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.395+0000 7f430cbbe640 1 Processor -- start 2024-04-25T15:20:47.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.395+0000 7f430cbbe640 1 -- start start 2024-04-25T15:20:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430cbbe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43080b4df0 0x7f4308156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430cbbe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 0x7f4308155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430cbbe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4308162890 0x7f4308153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430cbbe640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f43080d2830 con 0x7f4308162110 2024-04-25T15:20:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430cbbe640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43080d2030 con 0x7f43080b4df0 2024-04-25T15:20:47.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430cbbe640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f43080ceb10 con 0x7f4308162890 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 0x7f4308155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 0x7f4308155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56192/0 (socket says 172.21.15.146:56192) 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 -- 172.21.15.146:0/919059407 learned_addr learned my addr 172.21.15.146:0/919059407 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430659c640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43080b4df0 0x7f4308156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f4306d9d640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4308162890 0x7f4308153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 -- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4308162890 msgr2=0x7f4308153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4308162890 0x7f4308153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 -- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43080b4df0 msgr2=0x7f4308156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43080b4df0 0x7f4308156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f42fffff640 1 -- 172.21.15.146:0/919059407 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4308156060 con 0x7f4308162110 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.396+0000 7f430659c640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43080b4df0 0x7f4308156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.397+0000 7f42fffff640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 0x7f4308155b20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f42f00097d0 tx=0x7f42f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.397+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0037040 con 0x7f4308162110 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.397+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f0004230 con 0x7f4308162110 2024-04-25T15:20:47.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.397+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4308159210 con 0x7f4308162110 2024-04-25T15:20:47.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.397+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0006e10 con 0x7f4308162110 2024-04-25T15:20:47.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.397+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43081597d0 con 0x7f4308162110 2024-04-25T15:20:47.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.398+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f430806a0e0 con 0x7f4308162110 2024-04-25T15:20:47.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.399+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f42f0003610 con 0x7f4308162110 2024-04-25T15:20:47.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.399+0000 7f42feffd640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42e80744e0 0x7f42e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.400+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f42f00b64b0 con 0x7f4308162110 2024-04-25T15:20:47.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.400+0000 7f430659c640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42e80744e0 0x7f42e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.400+0000 7f430659c640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42e80744e0 0x7f42e80769f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f42f8001b30 tx=0x7f42f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.405+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42f0080850 con 0x7f4308162110 2024-04-25T15:20:47.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.601+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43080b61f0 con 0x7f4308162110 2024-04-25T15:20:47.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.603+0000 7f42feffd640 1 -- 172.21.15.146:0/919059407 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f42f0080150 con 0x7f4308162110 2024-04-25T15:20:47.604 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:47.604 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42e80744e0 msgr2=0x7f42e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42e80744e0 0x7f42e80769f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f42f8001b30 tx=0x7f42f8009040 comp rx=0 tx=0).stop 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 msgr2=0x7f4308155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 0x7f4308155b20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f42f00097d0 tx=0x7f42f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 shutdown_connections 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42e80744e0 0x7f42e80769f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4308162890 0x7f4308153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4308162110 0x7f4308155b20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 --2- 172.21.15.146:0/919059407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43080b4df0 0x7f4308156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 >> 172.21.15.146:0/919059407 conn(0x7f43080b52e0 msgr2=0x7f43080b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:47.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 shutdown_connections 2024-04-25T15:20:47.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.605+0000 7f430cbbe640 1 -- 172.21.15.146:0/919059407 wait complete. 2024-04-25T15:20:47.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:47.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:47.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.728+0000 7f62404f9640 1 Processor -- start 2024-04-25T15:20:47.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.728+0000 7f62404f9640 1 -- start start 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f62404f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f62404f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f62380bb450 0x7f62380d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f62404f9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62380d0de0 0x7f62380b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f62404f9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f623806dea0 con 0x7f62380d0de0 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f62404f9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f62380b5230 con 0x7f62380bb450 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f62404f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f623806c8c0 con 0x7f6238163c00 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62380d0de0 0x7f62380b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62380d0de0 0x7f62380b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56208/0 (socket says 172.21.15.146:56208) 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 -- 172.21.15.146:0/3543304785 learned_addr learned my addr 172.21.15.146:0/3543304785 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 -- 172.21.15.146:0/3543304785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 msgr2=0x7f623806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 --2- 172.21.15.146:0/3543304785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 -- 172.21.15.146:0/3543304785 >> v2:172.21.15.177:3300/0 conn(0x7f62380bb450 msgr2=0x7f62380d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623da94640 1 --2- 172.21.15.146:0/3543304785 >> v2:172.21.15.177:3300/0 conn(0x7f62380bb450 0x7f62380d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 --2- 172.21.15.146:0/3543304785 >> v2:172.21.15.177:3300/0 conn(0x7f62380bb450 0x7f62380d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623ea96640 1 -- 172.21.15.146:0/3543304785 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623806cba0 con 0x7f62380d0de0 2024-04-25T15:20:47.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.730+0000 7f623da94640 1 --2- 172.21.15.146:0/3543304785 >> v2:172.21.15.177:3300/0 conn(0x7f62380bb450 0x7f62380d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:47.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.731+0000 7f623ea96640 1 --2- 172.21.15.146:0/3543304785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f62380b8fb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f623808de60 tx=0x7f623002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e21b967f7cb03f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.731+0000 7f623d293640 1 -- 172.21.15.146:0/3543304785 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230037070 con 0x7f62380d0de0 2024-04-25T15:20:47.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.731+0000 7f623d293640 1 -- 172.21.15.146:0/3543304785 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6230032440 con 0x7f62380d0de0 2024-04-25T15:20:47.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.731+0000 7f623d293640 1 -- 172.21.15.146:0/3543304785 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230032810 con 0x7f62380d0de0 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 -- 172.21.15.146:0/3543304785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 msgr2=0x7f62380b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 --2- 172.21.15.146:0/3543304785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f62380b8fb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f623808de60 tx=0x7f623002a140 comp rx=0 tx=0).stop 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 -- 172.21.15.146:0/3543304785 shutdown_connections 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 --2- 172.21.15.146:0/3543304785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f62380b8fb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 --2- 172.21.15.146:0/3543304785 >> v2:172.21.15.177:3300/0 conn(0x7f62380bb450 0x7f62380d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 --2- 172.21.15.146:0/3543304785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 -- 172.21.15.146:0/3543304785 >> 172.21.15.146:0/3543304785 conn(0x7f62380b7970 msgr2=0x7f62380b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 -- 172.21.15.146:0/3543304785 shutdown_connections 2024-04-25T15:20:47.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.732+0000 7f62404f9640 1 -- 172.21.15.146:0/3543304785 wait complete. 2024-04-25T15:20:47.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.734+0000 7f62404f9640 1 Processor -- start 2024-04-25T15:20:47.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.734+0000 7f62404f9640 1 -- start start 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f62404f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62380bb450 0x7f623815e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f62404f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f623814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f62404f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623814ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f62404f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f623806c550 con 0x7f62380d0de0 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f62404f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f623806d160 con 0x7f62380bb450 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f62404f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f62380bb860 con 0x7f6238163c00 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f623814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f623814b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56212/0 (socket says 172.21.15.146:56212) 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 -- 172.21.15.146:0/1177327292 learned_addr learned my addr 172.21.15.146:0/1177327292 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623e295640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62380bb450 0x7f623815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623ea96640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623814ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 -- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 msgr2=0x7f623814ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623814ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 -- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62380bb450 msgr2=0x7f623815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62380bb450 0x7f623815e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623da94640 1 -- 172.21.15.146:0/1177327292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623815e5a0 con 0x7f62380d0de0 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.735+0000 7f623ea96640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623814ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.736+0000 7f623da94640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f623814b780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f622800c7f0 tx=0x7f622800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.736+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228015070 con 0x7f62380d0de0 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.736+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6228007820 con 0x7f62380d0de0 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.736+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228007bf0 con 0x7f62380d0de0 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.736+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6238158210 con 0x7f62380d0de0 2024-04-25T15:20:47.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.736+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62381587d0 con 0x7f62380d0de0 2024-04-25T15:20:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.737+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6238157f80 con 0x7f62380d0de0 2024-04-25T15:20:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.737+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6228018070 con 0x7f62380d0de0 2024-04-25T15:20:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.738+0000 7f622effd640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f621c0744b0 0x7f621c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.738+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f62280956d0 con 0x7f62380d0de0 2024-04-25T15:20:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.738+0000 7f623e295640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f621c0744b0 0x7f621c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:47.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.738+0000 7f623e295640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f621c0744b0 0x7f621c0769c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62380b5130 tx=0x7f623400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.742+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f623815e5a0 con 0x7f62380d0de0 2024-04-25T15:20:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.917+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f623806a0e0 con 0x7f62380d0de0 2024-04-25T15:20:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.917+0000 7f622effd640 1 -- 172.21.15.146:0/1177327292 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f622805f420 con 0x7f62380d0de0 2024-04-25T15:20:47.918 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f621c0744b0 msgr2=0x7f621c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f621c0744b0 0x7f621c0769c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62380b5130 tx=0x7f623400d040 comp rx=0 tx=0).stop 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 msgr2=0x7f623814b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f623814b780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f622800c7f0 tx=0x7f622800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 shutdown_connections 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f621c0744b0 0x7f621c0769c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6238163c00 0x7f623814ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62380d0de0 0x7f623814b780 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 --2- 172.21.15.146:0/1177327292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62380bb450 0x7f623815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:47.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 >> 172.21.15.146:0/1177327292 conn(0x7f62380b7970 msgr2=0x7f6238163210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:47.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 shutdown_connections 2024-04-25T15:20:47.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:47.919+0000 7f62404f9640 1 -- 172.21.15.146:0/1177327292 wait complete. 2024-04-25T15:20:47.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:48.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.043+0000 7f115d669640 1 Processor -- start 2024-04-25T15:20:48.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.043+0000 7f115d669640 1 -- start start 2024-04-25T15:20:48.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f115d669640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580d1140 0x7f11580b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f115d669640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11580b9520 con 0x7f11580b9110 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f115d669640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115806c5f0 con 0x7f11580b4c30 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f115d669640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f115806cc00 con 0x7f11580d1140 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f11577fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f11580b9110 0x7f115806d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34742/0 (socket says 172.21.15.146:34742) 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f11577fe640 1 -- 172.21.15.146:0/3622334257 learned_addr learned my addr 172.21.15.146:0/3622334257 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035862378 0 0) 0x7f11580b9520 con 0x7f11580b9110 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f11567fc640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580d1140 0x7f11580b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1140003520 con 0x7f11580b9110 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2170307193 0 0) 0x7f1140003520 con 0x7f11580b9110 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580d1140 msgr2=0x7f11580b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.045+0000 7f1155ffb640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580d1140 0x7f11580b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 >> v1:172.21.15.177:6789/0 conn(0x7f11580b4c30 legacy=0x7f11580d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11581647b0 con 0x7f11580b9110 2024-04-25T15:20:48.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f1150001670 con 0x7f11580b9110 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11567fc640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580d1140 0x7f11580b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 >> v1:172.21.15.146:6789/0 conn(0x7f11580b9110 legacy=0x7f115806d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11400039e0 0x7f1140003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1140004350 0x7f1140004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1140005030 0x7f114000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f115806cc00 con 0x7f1140005030 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f114000bf00 con 0x7f1140004350 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f114000c100 con 0x7f11400039e0 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11577fe640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1140005030 0x7f114000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f1156ffd640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11400039e0 0x7f1140003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11567fc640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1140004350 0x7f1140004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11577fe640 1 -- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11400039e0 msgr2=0x7f1140003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11577fe640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11400039e0 0x7f1140003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11577fe640 1 -- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1140004350 msgr2=0x7f1140004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11577fe640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1140004350 0x7f1140004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.046+0000 7f11577fe640 1 -- 172.21.15.146:0/3622334257 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1140003520 con 0x7f1140005030 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.047+0000 7f11577fe640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1140005030 0x7f114000b9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f115000c210 tx=0x7f1140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a86ef095753a42b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.047+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150007d30 con 0x7f1140005030 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.047+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11500058d0 con 0x7f1140005030 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.047+0000 7f1155ffb640 1 -- 172.21.15.146:0/3622334257 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150005ce0 con 0x7f1140005030 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f115d669640 1 -- 172.21.15.146:0/3622334257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1140005030 msgr2=0x7f114000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f115d669640 1 --2- 172.21.15.146:0/3622334257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1140005030 0x7f114000b9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f115000c210 tx=0x7f1140005830 comp rx=0 tx=0).stop 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f11577fe640 1 -- 172.21.15.146:0/3622334257 reap_dead start 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f115d669640 1 -- 172.21.15.146:0/3622334257 shutdown_connections 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f115d669640 1 -- 172.21.15.146:0/3622334257 >> 172.21.15.146:0/3622334257 conn(0x7f11580b7970 msgr2=0x7f11580b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f115d669640 1 -- 172.21.15.146:0/3622334257 shutdown_connections 2024-04-25T15:20:48.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.048+0000 7f115d669640 1 -- 172.21.15.146:0/3622334257 wait complete. 2024-04-25T15:20:48.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.050+0000 7f115d669640 1 Processor -- start 2024-04-25T15:20:48.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.050+0000 7f115d669640 1 -- start start 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f115d669640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580ca630 0x7f115815a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f115d669640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11580ced30 0x7f1158162670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f115d669640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 0x7f1158162bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f115d669640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f11580d16b0 con 0x7f11580cf140 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f115d669640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1158164a90 con 0x7f11580ced30 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f115d669640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f11580d1140 con 0x7f11580ca630 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 0x7f1158162bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 0x7f1158162bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56220/0 (socket says 172.21.15.146:56220) 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 -- 172.21.15.146:0/2200121624 learned_addr learned my addr 172.21.15.146:0/2200121624 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f1156ffd640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580ca630 0x7f115815a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11567fc640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11580ced30 0x7f1158162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 -- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580ca630 msgr2=0x7f115815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580ca630 0x7f115815a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 -- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11580ced30 msgr2=0x7f1158162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11580ced30 0x7f1158162670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.051+0000 7f11577fe640 1 -- 172.21.15.146:0/2200121624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11581f6150 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.052+0000 7f11577fe640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 0x7f1158162bb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f115002adf0 tx=0x7f1158164980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.052+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150007d30 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.052+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1150042070 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.052+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11581f6410 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.052+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150004c30 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.052+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581f69d0 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.053+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115806a0e0 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.054+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f115003b070 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.054+0000 7f113f7fe640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f11440744b0 0x7f11440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.054+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f11500b4c00 con 0x7f11580cf140 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.054+0000 7f1156ffd640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f11440744b0 0x7f11440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.055+0000 7f1156ffd640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f11440744b0 0x7f11440769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1158163890 tx=0x7f1148009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.059+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11500bb080 con 0x7f11580cf140 2024-04-25T15:20:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.234+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f11580b6ba0 con 0x7f11580cf140 2024-04-25T15:20:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.234+0000 7f113f7fe640 1 -- 172.21.15.146:0/2200121624 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f115007e9e0 con 0x7f11580cf140 2024-04-25T15:20:48.235 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f11440744b0 msgr2=0x7f11440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f11440744b0 0x7f11440769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1158163890 tx=0x7f1148009040 comp rx=0 tx=0).stop 2024-04-25T15:20:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 msgr2=0x7f1158162bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 0x7f1158162bb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f115002adf0 tx=0x7f1158164980 comp rx=0 tx=0).stop 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 shutdown_connections 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f11440744b0 0x7f11440769c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11580cf140 0x7f1158162bb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11580ced30 0x7f1158162670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 --2- 172.21.15.146:0/2200121624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11580ca630 0x7f115815a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 >> 172.21.15.146:0/2200121624 conn(0x7f11580b7970 msgr2=0x7f115815f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 shutdown_connections 2024-04-25T15:20:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.236+0000 7f115d669640 1 -- 172.21.15.146:0/2200121624 wait complete. 2024-04-25T15:20:48.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:48.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.359+0000 7fe2c2d1c640 1 Processor -- start 2024-04-25T15:20:48.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.359+0000 7fe2c2d1c640 1 -- start start 2024-04-25T15:20:48.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c2d1c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe2bc163c00 0x7fe2bc06d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c2d1c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0bb450 0x7fe2bc0d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c2d1c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2bc06dea0 con 0x7fe2bc0d0de0 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c2d1c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe2bc0b5230 con 0x7fe2bc163c00 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c2d1c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe2bc06c8c0 con 0x7fe2bc0bb450 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c0ab8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe2bc163c00 0x7fe2bc06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c0ab8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe2bc163c00 0x7fe2bc06d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56232/0 (socket says 172.21.15.146:56232) 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c12b9640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe2bc0d0de0 0x7fe2bc0b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34754/0 (socket says 172.21.15.146:34754) 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.360+0000 7fe2c0ab8640 1 -- 172.21.15.146:0/2231360841 learned_addr learned my addr 172.21.15.146:0/2231360841 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2b3fff640 1 --2- 172.21.15.146:0/2231360841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0bb450 0x7fe2bc0d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2c0ab8640 1 -- 172.21.15.146:0/2231360841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0bb450 msgr2=0x7fe2bc0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2c0ab8640 1 --2- 172.21.15.146:0/2231360841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0bb450 0x7fe2bc0d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2c0ab8640 1 -- 172.21.15.146:0/2231360841 >> v1:172.21.15.146:6789/0 conn(0x7fe2bc0d0de0 legacy=0x7fe2bc0b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2c0ab8640 1 -- 172.21.15.146:0/2231360841 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2bc06cba0 con 0x7fe2bc163c00 2024-04-25T15:20:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2b37fe640 1 -- 172.21.15.146:0/2231360841 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4158048316 0 0) 0x7fe2bc06dea0 con 0x7fe2bc0d0de0 2024-04-25T15:20:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2c0ab8640 1 --2- 172.21.15.146:0/2231360841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc06d960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009870 tx=0x7fe2b802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70752c647797fe26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.361+0000 7fe2b37fe640 1 -- 172.21.15.146:0/2231360841 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8037070 con 0x7fe2bc163c00 2024-04-25T15:20:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2b37fe640 1 -- 172.21.15.146:0/2231360841 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b8032440 con 0x7fe2bc163c00 2024-04-25T15:20:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2b37fe640 1 -- 172.21.15.146:0/2231360841 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8032810 con 0x7fe2bc163c00 2024-04-25T15:20:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/2231360841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 msgr2=0x7fe2bc06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/2231360841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc06d960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009870 tx=0x7fe2b802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/2231360841 shutdown_connections 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/2231360841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0bb450 0x7fe2bc0d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/2231360841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc06d960 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/2231360841 >> 172.21.15.146:0/2231360841 conn(0x7fe2bc0b7970 msgr2=0x7fe2bc0b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/2231360841 shutdown_connections 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.362+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/2231360841 wait complete. 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.364+0000 7fe2c2d1c640 1 Processor -- start 2024-04-25T15:20:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.364+0000 7fe2c2d1c640 1 -- start start 2024-04-25T15:20:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.365+0000 7fe2c2d1c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2bc0bb450 0x7fe2bc162f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.365+0000 7fe2c2d1c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0d0de0 0x7fe2bc15ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.365+0000 7fe2c2d1c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc1f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.365+0000 7fe2c2d1c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe2bc06c550 con 0x7fe2bc163c00 2024-04-25T15:20:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.365+0000 7fe2c2d1c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe2bc06d160 con 0x7fe2bc0bb450 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.365+0000 7fe2c2d1c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe2bc0bb860 con 0x7fe2bc0d0de0 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc1f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc1f68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56236/0 (socket says 172.21.15.146:56236) 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 -- 172.21.15.146:0/1328671333 learned_addr learned my addr 172.21.15.146:0/1328671333 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2b3fff640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0d0de0 0x7fe2bc15ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c0ab8640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2bc0bb450 0x7fe2bc162f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 -- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0d0de0 msgr2=0x7fe2bc15ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0d0de0 0x7fe2bc15ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 -- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2bc0bb450 msgr2=0x7fe2bc162f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2bc0bb450 0x7fe2bc162f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 -- 172.21.15.146:0/1328671333 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2bc162030 con 0x7fe2bc163c00 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2b3fff640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0d0de0 0x7fe2bc15ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c0ab8640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2bc0bb450 0x7fe2bc162f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c12b9640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc1f68f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400c6d0 tx=0x7fe2b400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4018070 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b4004d60 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2bc163450 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400a880 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.366+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2bc161a70 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.367+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2bc1636b0 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.368+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe2b401b070 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.368+0000 7fe2b17fa640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe2a80744e0 0x7fe2a80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.368+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fe2b4094d00 con 0x7fe2bc163c00 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.368+0000 7fe2c0ab8640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe2a80744e0 0x7fe2a80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.369+0000 7fe2c0ab8640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe2a80744e0 0x7fe2a80769f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8034000 tx=0x7fe2b802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.373+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2b405f130 con 0x7fe2bc163c00 2024-04-25T15:20:48.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.550+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe2bc15a890 con 0x7fe2bc163c00 2024-04-25T15:20:48.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.550+0000 7fe2b17fa640 1 -- 172.21.15.146:0/1328671333 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fe2b4067120 con 0x7fe2bc163c00 2024-04-25T15:20:48.551 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:48.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe2a80744e0 msgr2=0x7fe2a80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe2a80744e0 0x7fe2a80769f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8034000 tx=0x7fe2b802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:48.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 msgr2=0x7fe2bc1f68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc1f68f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400c6d0 tx=0x7fe2b400cb90 comp rx=0 tx=0).stop 2024-04-25T15:20:48.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 shutdown_connections 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe2a80744e0 0x7fe2a80769f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2bc163c00 0x7fe2bc1f68f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe2bc0d0de0 0x7fe2bc15ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 --2- 172.21.15.146:0/1328671333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2bc0bb450 0x7fe2bc162f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 >> 172.21.15.146:0/1328671333 conn(0x7fe2bc0b7970 msgr2=0x7fe2bc0b9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 shutdown_connections 2024-04-25T15:20:48.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.552+0000 7fe2c2d1c640 1 -- 172.21.15.146:0/1328671333 wait complete. 2024-04-25T15:20:48.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:48.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.674+0000 7ff64d19e640 1 Processor -- start 2024-04-25T15:20:48.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.674+0000 7ff64d19e640 1 -- start start 2024-04-25T15:20:48.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64d19e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 0x7ff64806f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64d19e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 0x7ff6480cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64d19e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff64808dd40 con 0x7ff6480b6030 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64d19e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff64808e020 con 0x7ff6480b4cc0 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64d19e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff64808daf0 con 0x7ff64806fc20 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64759e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 0x7ff6480cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff63ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 0x7ff64806f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64759e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 0x7ff6480cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:48418/0 (socket says 172.21.15.146:48418) 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff63ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 0x7ff64806f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59316/0 (socket says 172.21.15.146:59316) 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff64759e640 1 -- 172.21.15.146:0/3761401330 learned_addr learned my addr 172.21.15.146:0/3761401330 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.675+0000 7ff646d9d640 1 --1- 172.21.15.146:0/3761401330 >> v1:172.21.15.177:6789/0 conn(0x7ff6480b6030 0x7ff6480d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:55788/0 (socket says 172.21.15.146:55788) 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.676+0000 7ff64759e640 1 -- 172.21.15.146:0/3761401330 >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 msgr2=0x7ff64806f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.676+0000 7ff64759e640 1 --2- 172.21.15.146:0/3761401330 >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 0x7ff64806f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.676+0000 7ff64759e640 1 -- 172.21.15.146:0/3761401330 >> v1:172.21.15.177:6789/0 conn(0x7ff6480b6030 legacy=0x7ff6480d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.676+0000 7ff64759e640 1 -- 172.21.15.146:0/3761401330 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6480d27b0 con 0x7ff64806fc20 2024-04-25T15:20:48.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.676+0000 7ff63ffff640 1 --2- 172.21.15.146:0/3761401330 >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 0x7ff64806f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64759e640 1 --2- 172.21.15.146:0/3761401330 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 0x7ff6480cdf90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff64002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca0593e2404cc50a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64659c640 1 -- 172.21.15.146:0/3761401330 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640036070 con 0x7ff64806fc20 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64659c640 1 -- 172.21.15.146:0/3761401330 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff640031600 con 0x7ff64806fc20 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64659c640 1 -- 172.21.15.146:0/3761401330 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6400319d0 con 0x7ff64806fc20 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 -- 172.21.15.146:0/3761401330 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 msgr2=0x7ff6480cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3761401330 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 0x7ff6480cdf90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff64002a680 comp rx=0 tx=0).stop 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 -- 172.21.15.146:0/3761401330 shutdown_connections 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3761401330 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff64806fc20 0x7ff6480cdf90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3761401330 >> v2:172.21.15.177:3300/0 conn(0x7ff6480b4cc0 0x7ff64806f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 -- 172.21.15.146:0/3761401330 >> 172.21.15.146:0/3761401330 conn(0x7ff6480b5430 msgr2=0x7ff6480b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.677+0000 7ff64d19e640 1 -- 172.21.15.146:0/3761401330 shutdown_connections 2024-04-25T15:20:48.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.678+0000 7ff64d19e640 1 -- 172.21.15.146:0/3761401330 wait complete. 2024-04-25T15:20:48.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.679+0000 7ff64d19e640 1 Processor -- start 2024-04-25T15:20:48.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.679+0000 7ff64d19e640 1 -- start start 2024-04-25T15:20:48.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64d19e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff64806fc20 0x7ff6480b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64d19e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6480b4cc0 0x7ff64809ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64d19e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 0x7ff6480c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64d19e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480d2a90 con 0x7ff6480b6030 2024-04-25T15:20:48.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64d19e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480d2490 con 0x7ff64806fc20 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64d19e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff6480d2c10 con 0x7ff6480b4cc0 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 0x7ff6480c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 0x7ff6480c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56238/0 (socket says 172.21.15.146:56238) 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 -- 172.21.15.146:0/3896896685 learned_addr learned my addr 172.21.15.146:0/3896896685 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff63ffff640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6480b4cc0 0x7ff64809ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff646d9d640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff64806fc20 0x7ff6480b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 -- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6480b4cc0 msgr2=0x7ff64809ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6480b4cc0 0x7ff64809ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 -- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff64806fc20 msgr2=0x7ff6480b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff64806fc20 0x7ff6480b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 -- 172.21.15.146:0/3896896685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64809f340 con 0x7ff6480b6030 2024-04-25T15:20:48.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.681+0000 7ff64759e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 0x7ff6480c8430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff640009f70 tx=0x7ff640004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.682+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640036050 con 0x7ff6480b6030 2024-04-25T15:20:48.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.682+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff640042070 con 0x7ff6480b6030 2024-04-25T15:20:48.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.682+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6480b30f0 con 0x7ff6480b6030 2024-04-25T15:20:48.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.682+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640009510 con 0x7ff6480b6030 2024-04-25T15:20:48.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.682+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6480b36b0 con 0x7ff6480b6030 2024-04-25T15:20:48.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.683+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64806a0e0 con 0x7ff6480b6030 2024-04-25T15:20:48.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.683+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff640007c50 con 0x7ff6480b6030 2024-04-25T15:20:48.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.684+0000 7ff63f7fe640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6280744e0 0x7ff6280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.684+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7ff6400b5d20 con 0x7ff6480b6030 2024-04-25T15:20:48.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.684+0000 7ff646d9d640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6280744e0 0x7ff6280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.684+0000 7ff646d9d640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6280744e0 0x7ff6280769f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff638007fd0 tx=0x7ff638007e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.688+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff64004b020 con 0x7ff6480b6030 2024-04-25T15:20:48.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.864+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff64806d480 con 0x7ff6480b6030 2024-04-25T15:20:48.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.865+0000 7ff63f7fe640 1 -- 172.21.15.146:0/3896896685 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7ff64007fa70 con 0x7ff6480b6030 2024-04-25T15:20:48.866 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6280744e0 msgr2=0x7ff6280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6280744e0 0x7ff6280769f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff638007fd0 tx=0x7ff638007e80 comp rx=0 tx=0).stop 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 msgr2=0x7ff6480c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 0x7ff6480c8430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff640009f70 tx=0x7ff640004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 shutdown_connections 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff6280744e0 0x7ff6280769f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6480b6030 0x7ff6480c8430 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6480b4cc0 0x7ff64809ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 --2- 172.21.15.146:0/3896896685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff64806fc20 0x7ff6480b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 >> 172.21.15.146:0/3896896685 conn(0x7ff6480b5430 msgr2=0x7ff64815f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 shutdown_connections 2024-04-25T15:20:48.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.867+0000 7ff64d19e640 1 -- 172.21.15.146:0/3896896685 wait complete. 2024-04-25T15:20:48.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.988+0000 7f13006d8640 1 Processor -- start 2024-04-25T15:20:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.988+0000 7f13006d8640 1 -- start start 2024-04-25T15:20:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f13006d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 0x7f12f80cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f13006d8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f80b4a60 con 0x7f12f80cce10 2024-04-25T15:20:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f13006d8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f80b4d40 con 0x7f12f80d19d0 2024-04-25T15:20:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f13006d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f12f80b5020 con 0x7f12f80cfbe0 2024-04-25T15:20:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fec75640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f12f80cce10 0x7f12f80cd220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34756/0 (socket says 172.21.15.146:34756) 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fec75640 1 -- 172.21.15.146:0/242846761 learned_addr learned my addr 172.21.15.146:0/242846761 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fe474640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 0x7f12f80cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779167201 0 0) 0x7f12f80b4a60 con 0x7f12f80cce10 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12e4003520 con 0x7f12f80cce10 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 190258481 0 0) 0x7f12e4003520 con 0x7f12f80cce10 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 msgr2=0x7f12f80cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 0x7f12f80cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 >> v1:172.21.15.177:6789/0 conn(0x7f12f80d19d0 legacy=0x7f12f80d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f80cead0 con 0x7f12f80cce10 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.990+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f12f0001670 con 0x7f12f80cce10 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 >> v1:172.21.15.146:6789/0 conn(0x7f12f80cce10 legacy=0x7f12f80cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e40039e0 0x7f12e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 0x7f12e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e4004d40 0x7f12e4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80b5020 con 0x7f12e4004d40 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12e400ba70 con 0x7f12e40039e0 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f12e400bc70 con 0x7f12f80cfbe0 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fec75640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e4004d40 0x7f12e4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fe474640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e40039e0 0x7f12e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fdc73640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 0x7f12e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fec75640 1 -- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 msgr2=0x7f12e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fec75640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cfbe0 0x7f12e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fec75640 1 -- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e40039e0 msgr2=0x7f12e4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fec75640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e40039e0 0x7f12e4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.991+0000 7f12fec75640 1 -- 172.21.15.146:0/242846761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e4003520 con 0x7f12e4004d40 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f12fec75640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e4004d40 0x7f12e4005150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f12f000c240 tx=0x7f12e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bd5f1184673fd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0007d30 con 0x7f12e4004d40 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f0005af0 con 0x7f12e4004d40 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f12fd472640 1 -- 172.21.15.146:0/242846761 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0007070 con 0x7f12e4004d40 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f13006d8640 1 -- 172.21.15.146:0/242846761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e4004d40 msgr2=0x7f12e4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f13006d8640 1 --2- 172.21.15.146:0/242846761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e4004d40 0x7f12e4005150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f12f000c240 tx=0x7f12e4005830 comp rx=0 tx=0).stop 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.992+0000 7f12fec75640 1 -- 172.21.15.146:0/242846761 reap_dead start 2024-04-25T15:20:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.993+0000 7f13006d8640 1 -- 172.21.15.146:0/242846761 shutdown_connections 2024-04-25T15:20:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.993+0000 7f13006d8640 1 -- 172.21.15.146:0/242846761 >> 172.21.15.146:0/242846761 conn(0x7f12f80b73b0 msgr2=0x7f12f80b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.993+0000 7f13006d8640 1 -- 172.21.15.146:0/242846761 shutdown_connections 2024-04-25T15:20:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.993+0000 7f13006d8640 1 -- 172.21.15.146:0/242846761 wait complete. 2024-04-25T15:20:49.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.994+0000 7f13006d8640 1 Processor -- start 2024-04-25T15:20:49.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.994+0000 7f13006d8640 1 -- start start 2024-04-25T15:20:49.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e4004d40 0x7f12f80c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cb240 0x7f12f8155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 0x7f12f80c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80cf3e0 con 0x7f12f81564c0 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80cf260 con 0x7f12e4004d40 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f12f80cf560 con 0x7f12f80cb240 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 0x7f12f80c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 0x7f12f80c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56252/0 (socket says 172.21.15.146:56252) 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 -- 172.21.15.146:0/620574095 learned_addr learned my addr 172.21.15.146:0/620574095 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fdc73640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cb240 0x7f12f8155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fe474640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e4004d40 0x7f12f80c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 -- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cb240 msgr2=0x7f12f8155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cb240 0x7f12f8155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 -- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e4004d40 msgr2=0x7f12f80c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e4004d40 0x7f12f80c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 -- 172.21.15.146:0/620574095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f80ca1a0 con 0x7f12f81564c0 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fdc73640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cb240 0x7f12f8155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fe474640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e4004d40 0x7f12f80c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12fec75640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 0x7f12f80c61f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f12f000c360 tx=0x7f12f80b4f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0043070 con 0x7f12f81564c0 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f0007280 con 0x7f12f81564c0 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f80ca480 con 0x7f12f81564c0 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0007d30 con 0x7f12f81564c0 2024-04-25T15:20:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.996+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f80caa40 con 0x7f12f81564c0 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.998+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f80d1de0 con 0x7f12f81564c0 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.998+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f12f0046070 con 0x7f12f81564c0 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.999+0000 7f12eeffd640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f12dc0744b0 0x7f12dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.999+0000 7f12fe474640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f12dc0744b0 0x7f12dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.999+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f12f00b4ca0 con 0x7f12f81564c0 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:48.999+0000 7f12fe474640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f12dc0744b0 0x7f12dc0769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f12f4007520 tx=0x7f12f400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.003+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f12f80d1de0 con 0x7f12f81564c0 2024-04-25T15:20:49.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.188+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f12f806a0e0 con 0x7f12f81564c0 2024-04-25T15:20:49.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.189+0000 7f12eeffd640 1 -- 172.21.15.146:0/620574095 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f12f007f0d0 con 0x7f12f81564c0 2024-04-25T15:20:49.189 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:49.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f12dc0744b0 msgr2=0x7f12dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f12dc0744b0 0x7f12dc0769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f12f4007520 tx=0x7f12f400b040 comp rx=0 tx=0).stop 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 msgr2=0x7f12f80c61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 0x7f12f80c61f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f12f000c360 tx=0x7f12f80b4f10 comp rx=0 tx=0).stop 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 shutdown_connections 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f12dc0744b0 0x7f12dc0769c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12f81564c0 0x7f12f80c61f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12f80cb240 0x7f12f8155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 --2- 172.21.15.146:0/620574095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12e4004d40 0x7f12f80c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 >> 172.21.15.146:0/620574095 conn(0x7f12f80b73b0 msgr2=0x7f12f80d2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 shutdown_connections 2024-04-25T15:20:49.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.191+0000 7f13006d8640 1 -- 172.21.15.146:0/620574095 wait complete. 2024-04-25T15:20:49.203 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:20:49.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:49.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-25T15:20:49.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:20:49.227 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:20:49.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:49.227 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T15:20:49.281 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:20:49.282 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:49.300 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:20:49.300 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:49.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:49.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.467+0000 7f89c7b0d640 1 Processor -- start 2024-04-25T15:20:49.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.467+0000 7f89c7b0d640 1 -- start start 2024-04-25T15:20:49.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.468+0000 7f89c7b0d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f89c006f120 0x7f89c009b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.468+0000 7f89c7b0d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89c00d0a40 0x7f89c00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.468+0000 7f89c7b0d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c00cfa90 con 0x7f89c00b3a10 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.468+0000 7f89c7b0d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f89c00d0050 con 0x7f89c00d0a40 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.468+0000 7f89c7b0d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f89c009bfb0 con 0x7f89c006f120 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89c00d0a40 0x7f89c00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89c00d0a40 0x7f89c00b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56262/0 (socket says 172.21.15.146:56262) 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 -- 172.21.15.146:0/744974520 learned_addr learned my addr 172.21.15.146:0/744974520 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c58a9640 1 --2- 172.21.15.146:0/744974520 >> v2:172.21.15.177:3300/0 conn(0x7f89c006f120 0x7f89c009b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 -- 172.21.15.146:0/744974520 >> v2:172.21.15.177:3300/0 conn(0x7f89c006f120 msgr2=0x7f89c009b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 --2- 172.21.15.146:0/744974520 >> v2:172.21.15.177:3300/0 conn(0x7f89c006f120 0x7f89c009b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 -- 172.21.15.146:0/744974520 >> v1:172.21.15.177:6789/0 conn(0x7f89c00b3a10 legacy=0x7f89c00d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.469+0000 7f89c60aa640 1 -- 172.21.15.146:0/744974520 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c00cfd70 con 0x7f89c00d0a40 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.470+0000 7f89c60aa640 1 --2- 172.21.15.146:0/744974520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c00d0a40 0x7f89c00b3e20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b5450 tx=0x7f89b802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44cc755a835c978 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.470+0000 7f89c48a7640 1 -- 172.21.15.146:0/744974520 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8036070 con 0x7f89c00d0a40 2024-04-25T15:20:49.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.470+0000 7f89c48a7640 1 -- 172.21.15.146:0/744974520 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b8031600 con 0x7f89c00d0a40 2024-04-25T15:20:49.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.470+0000 7f89c48a7640 1 -- 172.21.15.146:0/744974520 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80319d0 con 0x7f89c00d0a40 2024-04-25T15:20:49.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 -- 172.21.15.146:0/744974520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c00d0a40 msgr2=0x7f89c00b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/744974520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c00d0a40 0x7f89c00b3e20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f89c00b5450 tx=0x7f89b802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 -- 172.21.15.146:0/744974520 shutdown_connections 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/744974520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c00d0a40 0x7f89c00b3e20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/744974520 >> v2:172.21.15.177:3300/0 conn(0x7f89c006f120 0x7f89c009b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 -- 172.21.15.146:0/744974520 >> 172.21.15.146:0/744974520 conn(0x7f89c00b7530 msgr2=0x7f89c00b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 -- 172.21.15.146:0/744974520 shutdown_connections 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.471+0000 7f89c7b0d640 1 -- 172.21.15.146:0/744974520 wait complete. 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.473+0000 7f89c7b0d640 1 Processor -- start 2024-04-25T15:20:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.473+0000 7f89c7b0d640 1 -- start start 2024-04-25T15:20:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c7b0d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 0x7f89c0160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c7b0d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c00b3a10 0x7f89c015a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c7b0d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89c00d0a40 0x7f89c0162530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c7b0d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89c01603f0 con 0x7f89c006f120 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c7b0d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89c009b7f0 con 0x7f89c00b3a10 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c7b0d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f89c0160570 con 0x7f89c00d0a40 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 0x7f89c0160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 0x7f89c0160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56264/0 (socket says 172.21.15.146:56264) 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 -- 172.21.15.146:0/3651752106 learned_addr learned my addr 172.21.15.146:0/3651752106 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 -- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89c00d0a40 msgr2=0x7f89c0162530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89c00d0a40 0x7f89c0162530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 -- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c00b3a10 msgr2=0x7f89c015a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c50a8640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c00b3a10 0x7f89c015a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c00b3a10 0x7f89c015a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 -- 172.21.15.146:0/3651752106 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c00b1ec0 con 0x7f89c006f120 2024-04-25T15:20:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.474+0000 7f89c58a9640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 0x7f89c0160750 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00e8a0 tx=0x7f89bc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.475+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc017070 con 0x7f89c006f120 2024-04-25T15:20:49.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.475+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89bc00c940 con 0x7f89c006f120 2024-04-25T15:20:49.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.475+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c01612c0 con 0x7f89c006f120 2024-04-25T15:20:49.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.475+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc00cd30 con 0x7f89c006f120 2024-04-25T15:20:49.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.475+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c0161880 con 0x7f89c006f120 2024-04-25T15:20:49.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.476+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c006a0e0 con 0x7f89c006f120 2024-04-25T15:20:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.476+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f89bc01a070 con 0x7f89c006f120 2024-04-25T15:20:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.477+0000 7f89b67fc640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89a40744e0 0x7f89a40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.477+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f89bc0954e0 con 0x7f89c006f120 2024-04-25T15:20:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.477+0000 7f89c50a8640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89a40744e0 0x7f89a40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.477+0000 7f89c50a8640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89a40744e0 0x7f89a40769f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f89c00cbdb0 tx=0x7f89b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.481+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89bc05f880 con 0x7f89c006f120 2024-04-25T15:20:49.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.656+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89c00cc560 con 0x7f89c006f120 2024-04-25T15:20:49.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.657+0000 7f89b67fc640 1 -- 172.21.15.146:0/3651752106 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f89bc05f180 con 0x7f89c006f120 2024-04-25T15:20:49.658 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:49.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89a40744e0 msgr2=0x7f89a40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89a40744e0 0x7f89a40769f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f89c00cbdb0 tx=0x7f89b0009040 comp rx=0 tx=0).stop 2024-04-25T15:20:49.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 msgr2=0x7f89c0160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 0x7f89c0160750 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00e8a0 tx=0x7f89bc00ed60 comp rx=0 tx=0).stop 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 shutdown_connections 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89a40744e0 0x7f89a40769f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89c00d0a40 0x7f89c0162530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c00b3a10 0x7f89c015a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.659+0000 7f89c7b0d640 1 --2- 172.21.15.146:0/3651752106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89c006f120 0x7f89c0160750 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.660+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 >> 172.21.15.146:0/3651752106 conn(0x7f89c00b7530 msgr2=0x7f89c00d16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.660+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 shutdown_connections 2024-04-25T15:20:49.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.660+0000 7f89c7b0d640 1 -- 172.21.15.146:0/3651752106 wait complete. 2024-04-25T15:20:49.672 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:49.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:49.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.783+0000 7f3ce9140640 1 Processor -- start 2024-04-25T15:20:49.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.783+0000 7f3ce9140640 1 -- start start 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce9140640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ce40b2cd0 0x7f3ce40d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce9140640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce4071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce9140640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3ce4071b60 0x7f3ce408cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce9140640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce4071f70 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce9140640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce40d2260 con 0x7f3ce4071b60 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce9140640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce40d2540 con 0x7f3ce40d3300 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ce40b2cd0 0x7f3ce40d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ce40b2cd0 0x7f3ce40d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56280/0 (socket says 172.21.15.146:56280) 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 -- 172.21.15.146:0/3579505484 learned_addr learned my addr 172.21.15.146:0/3579505484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce359e640 1 --2- 172.21.15.146:0/3579505484 >> v2:172.21.15.177:3300/0 conn(0x7f3ce4071b60 0x7f3ce408cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce259c640 1 --2- 172.21.15.146:0/3579505484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce4071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 -- 172.21.15.146:0/3579505484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 msgr2=0x7f3ce4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 --2- 172.21.15.146:0/3579505484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce4071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 -- 172.21.15.146:0/3579505484 >> v2:172.21.15.177:3300/0 conn(0x7f3ce4071b60 msgr2=0x7f3ce408cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 --2- 172.21.15.146:0/3579505484 >> v2:172.21.15.177:3300/0 conn(0x7f3ce4071b60 0x7f3ce408cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce2d9d640 1 -- 172.21.15.146:0/3579505484 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce40d2820 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.785+0000 7f3ce359e640 1 --2- 172.21.15.146:0/3579505484 >> v2:172.21.15.177:3300/0 conn(0x7f3ce4071b60 0x7f3ce408cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.786+0000 7f3ce2d9d640 1 --2- 172.21.15.146:0/3579505484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40d1c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40744d0 tx=0x7f3cd402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b268fac74ad27a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.786+0000 7f3ce1d9b640 1 -- 172.21.15.146:0/3579505484 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4036070 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.786+0000 7f3ce1d9b640 1 -- 172.21.15.146:0/3579505484 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4031600 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.786+0000 7f3ce1d9b640 1 -- 172.21.15.146:0/3579505484 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40319d0 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 -- 172.21.15.146:0/3579505484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 msgr2=0x7f3ce40d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 --2- 172.21.15.146:0/3579505484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40d1c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40744d0 tx=0x7f3cd402a660 comp rx=0 tx=0).stop 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 -- 172.21.15.146:0/3579505484 shutdown_connections 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 --2- 172.21.15.146:0/3579505484 >> v2:172.21.15.177:3300/0 conn(0x7f3ce4071b60 0x7f3ce408cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 --2- 172.21.15.146:0/3579505484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce4071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 --2- 172.21.15.146:0/3579505484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40d1c80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 -- 172.21.15.146:0/3579505484 >> 172.21.15.146:0/3579505484 conn(0x7f3ce40b52e0 msgr2=0x7f3ce40b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 -- 172.21.15.146:0/3579505484 shutdown_connections 2024-04-25T15:20:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.787+0000 7f3ce9140640 1 -- 172.21.15.146:0/3579505484 wait complete. 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.789+0000 7f3ce9140640 1 Processor -- start 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.789+0000 7f3ce9140640 1 -- start start 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce9140640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ce4071b60 0x7f3ce41f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce9140640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce9140640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce40cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce9140640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40cfb70 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce9140640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40d2de0 con 0x7f3ce4071b60 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce9140640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce408dbf0 con 0x7f3ce40d3300 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56282/0 (socket says 172.21.15.146:56282) 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 -- 172.21.15.146:0/435141447 learned_addr learned my addr 172.21.15.146:0/435141447 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce2d9d640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ce4071b60 0x7f3ce41f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce359e640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce40cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 -- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 msgr2=0x7f3ce40cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce40cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 -- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ce4071b60 msgr2=0x7f3ce41f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ce4071b60 0x7f3ce41f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.790+0000 7f3ce259c640 1 -- 172.21.15.146:0/435141447 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce40c8510 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.791+0000 7f3ce259c640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40cc2a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c7f0 tx=0x7f3cd800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.791+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8015070 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.791+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd8004220 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.791+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce40c87f0 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.791+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8007620 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.791+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce40c8db0 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.792+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce40b72a0 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.792+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3cd80077c0 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.793+0000 7f3ccb7fe640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd00744b0 0x7f3cd00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:49.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.793+0000 7f3ce2d9d640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd00744b0 0x7f3cd00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:49.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.793+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f3cd8095720 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.793+0000 7f3ce2d9d640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd00744b0 0x7f3cd00769c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4033000 tx=0x7f3cd4009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:49.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.797+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cd805f470 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.973+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ce406a0e0 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.974+0000 7f3ccb7fe640 1 -- 172.21.15.146:0/435141447 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3cd8063770 con 0x7f3ce40b2cd0 2024-04-25T15:20:49.975 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd00744b0 msgr2=0x7f3cd00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd00744b0 0x7f3cd00769c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4033000 tx=0x7f3cd4009000 comp rx=0 tx=0).stop 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 msgr2=0x7f3ce40cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40cc2a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c7f0 tx=0x7f3cd800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 shutdown_connections 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3cd00744b0 0x7f3cd00769c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ce40d3300 0x7f3ce40cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ce40b2cd0 0x7f3ce40cc2a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 --2- 172.21.15.146:0/435141447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ce4071b60 0x7f3ce41f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 >> 172.21.15.146:0/435141447 conn(0x7f3ce40b52e0 msgr2=0x7f3ce40b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:49.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 shutdown_connections 2024-04-25T15:20:49.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:49.976+0000 7f3ce9140640 1 -- 172.21.15.146:0/435141447 wait complete. 2024-04-25T15:20:49.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:50.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.097+0000 7fc0790af640 1 Processor -- start 2024-04-25T15:20:50.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.097+0000 7fc0790af640 1 -- start start 2024-04-25T15:20:50.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc0790af640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 0x7fc0740cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc0790af640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc0740d2440 0x7fc074071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc0790af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0740cf7d0 con 0x7fc0740d1920 2024-04-25T15:20:50.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc0790af640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc07408da90 con 0x7fc0740d2440 2024-04-25T15:20:50.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc0790af640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc0740d0070 con 0x7fc0740b2cd0 2024-04-25T15:20:50.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc07259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc0740d1920 0x7fc0740d1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34758/0 (socket says 172.21.15.146:34758) 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc07259c640 1 -- 172.21.15.146:0/2301460804 learned_addr learned my addr 172.21.15.146:0/2301460804 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc07359e640 1 --2- 172.21.15.146:0/2301460804 >> v2:172.21.15.177:3300/0 conn(0x7fc0740d2440 0x7fc074071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3417611791 0 0) 0x7fc0740cf7d0 con 0x7fc0740d1920 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc072d9d640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 0x7fc0740cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc060003520 con 0x7fc0740d1920 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1228058598 0 0) 0x7fc060003520 con 0x7fc0740d1920 2024-04-25T15:20:50.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 msgr2=0x7fc0740cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 0x7fc0740cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 >> v2:172.21.15.177:3300/0 conn(0x7fc0740d2440 msgr2=0x7fc074071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 --2- 172.21.15.146:0/2301460804 >> v2:172.21.15.177:3300/0 conn(0x7fc0740d2440 0x7fc074071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0740cfab0 con 0x7fc0740d1920 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fc05c001670 con 0x7fc0740d1920 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 >> v1:172.21.15.146:6789/0 conn(0x7fc0740d1920 legacy=0x7fc0740d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 0x7fc0600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc060003f30 0x7fc060004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc060004d40 0x7fc060005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0740d0070 con 0x7fc060003f30 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc06000ba70 con 0x7fc060004d40 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc06000bc40 con 0x7fc0740b2cd0 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.099+0000 7fc07259c640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc060003f30 0x7fc060004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc072d9d640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 0x7fc0600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07359e640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc060004d40 0x7fc060005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07259c640 1 -- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 msgr2=0x7fc0600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07259c640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0740b2cd0 0x7fc0600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07259c640 1 -- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc060004d40 msgr2=0x7fc060005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07259c640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc060004d40 0x7fc060005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07259c640 1 -- 172.21.15.146:0/2301460804 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc060003520 con 0x7fc060003f30 2024-04-25T15:20:50.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.100+0000 7fc07259c640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc060003f30 0x7fc060004800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc05c001050 tx=0x7fc060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1241e120fd3f4aeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c02e070 con 0x7fc060003f30 2024-04-25T15:20:50.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05c005af0 con 0x7fc060003f30 2024-04-25T15:20:50.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc071d9b640 1 -- 172.21.15.146:0/2301460804 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c007d70 con 0x7fc060003f30 2024-04-25T15:20:50.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc0790af640 1 -- 172.21.15.146:0/2301460804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc060003f30 msgr2=0x7fc060004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc0790af640 1 --2- 172.21.15.146:0/2301460804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc060003f30 0x7fc060004800 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc05c001050 tx=0x7fc060005830 comp rx=0 tx=0).stop 2024-04-25T15:20:50.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc07359e640 1 -- 172.21.15.146:0/2301460804 reap_dead start 2024-04-25T15:20:50.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc0790af640 1 -- 172.21.15.146:0/2301460804 shutdown_connections 2024-04-25T15:20:50.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc0790af640 1 -- 172.21.15.146:0/2301460804 >> 172.21.15.146:0/2301460804 conn(0x7fc0740b52e0 msgr2=0x7fc0740b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:50.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc0790af640 1 -- 172.21.15.146:0/2301460804 shutdown_connections 2024-04-25T15:20:50.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.101+0000 7fc0790af640 1 -- 172.21.15.146:0/2301460804 wait complete. 2024-04-25T15:20:50.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.103+0000 7fc0790af640 1 Processor -- start 2024-04-25T15:20:50.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.103+0000 7fc0790af640 1 -- start start 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc060003f30 0x7fc074152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 0x7fc074160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0741610a0 0x7fc074072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0740cf0f0 con 0x7fc0740cbac0 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc0740717e0 con 0x7fc0741610a0 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc0740cf270 con 0x7fc060003f30 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 0x7fc074160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 0x7fc074160b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56292/0 (socket says 172.21.15.146:56292) 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 -- 172.21.15.146:0/78217718 learned_addr learned my addr 172.21.15.146:0/78217718 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc072d9d640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc060003f30 0x7fc074152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07359e640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0741610a0 0x7fc074072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 -- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc060003f30 msgr2=0x7fc074152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc060003f30 0x7fc074152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 -- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0741610a0 msgr2=0x7fc074072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0741610a0 0x7fc074072110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 -- 172.21.15.146:0/78217718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0741614b0 con 0x7fc0740cbac0 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc072d9d640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc060003f30 0x7fc074152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc07259c640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 0x7fc074160b60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00c300 tx=0x7fc05c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c042070 con 0x7fc0740cbac0 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05c007480 con 0x7fc0740cbac0 2024-04-25T15:20:50.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.105+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0740caa20 con 0x7fc0740cbac0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.106+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c007d50 con 0x7fc0740cbac0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.106+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0740cafe0 con 0x7fc0740cbac0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.106+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07406a0e0 con 0x7fc0740cbac0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.107+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc05c045070 con 0x7fc0740cbac0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.108+0000 7fc06b7fe640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0540744e0 0x7fc0540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.108+0000 7fc072d9d640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0540744e0 0x7fc0540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.108+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fc05c0b60b0 con 0x7fc0740cbac0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.108+0000 7fc072d9d640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0540744e0 0x7fc0540769f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc064009670 tx=0x7fc06400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.113+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0741614b0 con 0x7fc0740cbac0 2024-04-25T15:20:50.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.285+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0741531a0 con 0x7fc0740cbac0 2024-04-25T15:20:50.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.286+0000 7fc06b7fe640 1 -- 172.21.15.146:0/78217718 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc05c005ac0 con 0x7fc0740cbac0 2024-04-25T15:20:50.287 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0540744e0 msgr2=0x7fc0540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0540744e0 0x7fc0540769f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc064009670 tx=0x7fc06400b040 comp rx=0 tx=0).stop 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 msgr2=0x7fc074160b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 0x7fc074160b60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00c300 tx=0x7fc05c000fe0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 shutdown_connections 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0540744e0 0x7fc0540769f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0741610a0 0x7fc074072110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0740cbac0 0x7fc074160b60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 --2- 172.21.15.146:0/78217718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc060003f30 0x7fc074152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 >> 172.21.15.146:0/78217718 conn(0x7fc0740b52e0 msgr2=0x7fc0740b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:50.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 shutdown_connections 2024-04-25T15:20:50.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.288+0000 7fc0790af640 1 -- 172.21.15.146:0/78217718 wait complete. 2024-04-25T15:20:50.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:50.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.411+0000 7fe45f0f9640 1 Processor -- start 2024-04-25T15:20:50.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.411+0000 7fe45f0f9640 1 -- start start 2024-04-25T15:20:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45f0f9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe45815e070 0x7fe4580cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45f0f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe4580d2680 0x7fe4580d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45f0f9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4580d2fd0 con 0x7fe45809bd80 2024-04-25T15:20:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45f0f9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe4580ce920 con 0x7fe45815e070 2024-04-25T15:20:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45f0f9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe4580cec00 con 0x7fe4580d2680 2024-04-25T15:20:50.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45ce95640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe45815e070 0x7fe4580cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe457fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe45809bd80 0x7fe4580ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:34760/0 (socket says 172.21.15.146:34760) 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe457fff640 1 -- 172.21.15.146:0/4093719494 learned_addr learned my addr 172.21.15.146:0/4093719494 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45d696640 1 --2- 172.21.15.146:0/4093719494 >> v2:172.21.15.177:3300/0 conn(0x7fe4580d2680 0x7fe4580d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45ce95640 1 -- 172.21.15.146:0/4093719494 >> v2:172.21.15.177:3300/0 conn(0x7fe4580d2680 msgr2=0x7fe4580d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45ce95640 1 --2- 172.21.15.146:0/4093719494 >> v2:172.21.15.177:3300/0 conn(0x7fe4580d2680 0x7fe4580d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45ce95640 1 -- 172.21.15.146:0/4093719494 >> v1:172.21.15.146:6789/0 conn(0x7fe45809bd80 legacy=0x7fe4580ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe45ce95640 1 -- 172.21.15.146:0/4093719494 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4580ceee0 con 0x7fe45815e070 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.412+0000 7fe4577fe640 1 -- 172.21.15.146:0/4093719494 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002518119 0 0) 0x7fe44c001110 con 0x7fe45809bd80 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.413+0000 7fe45ce95640 1 --2- 172.21.15.146:0/4093719494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe45815e070 0x7fe4580cf9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe4580b55f0 tx=0x7fe44802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1beb5aea018e63ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.413+0000 7fe4577fe640 1 -- 172.21.15.146:0/4093719494 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448037070 con 0x7fe45815e070 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.413+0000 7fe4577fe640 1 -- 172.21.15.146:0/4093719494 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe448032440 con 0x7fe45815e070 2024-04-25T15:20:50.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.413+0000 7fe4577fe640 1 -- 172.21.15.146:0/4093719494 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448032810 con 0x7fe45815e070 2024-04-25T15:20:50.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 -- 172.21.15.146:0/4093719494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe45815e070 msgr2=0x7fe4580cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/4093719494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe45815e070 0x7fe4580cf9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe4580b55f0 tx=0x7fe44802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 -- 172.21.15.146:0/4093719494 shutdown_connections 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/4093719494 >> v2:172.21.15.177:3300/0 conn(0x7fe4580d2680 0x7fe4580d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/4093719494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe45815e070 0x7fe4580cf9c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 -- 172.21.15.146:0/4093719494 >> 172.21.15.146:0/4093719494 conn(0x7fe4580b7400 msgr2=0x7fe4580b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 -- 172.21.15.146:0/4093719494 shutdown_connections 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.414+0000 7fe45f0f9640 1 -- 172.21.15.146:0/4093719494 wait complete. 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.416+0000 7fe45f0f9640 1 Processor -- start 2024-04-25T15:20:50.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.416+0000 7fe45f0f9640 1 -- start start 2024-04-25T15:20:50.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45f0f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe45809bd80 0x7fe4580cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45f0f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4580d2680 0x7fe458157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45f0f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 0x7fe45806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45f0f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580cff00 con 0x7fe4581e7830 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45f0f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580cf4a0 con 0x7fe4580d2680 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45f0f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe45806cde0 con 0x7fe45809bd80 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45d696640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 0x7fe45806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45d696640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 0x7fe45806d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56306/0 (socket says 172.21.15.146:56306) 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45d696640 1 -- 172.21.15.146:0/2566491676 learned_addr learned my addr 172.21.15.146:0/2566491676 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe45ce95640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe45809bd80 0x7fe4580cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.417+0000 7fe457fff640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4580d2680 0x7fe458157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45d696640 1 -- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe45809bd80 msgr2=0x7fe4580cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45d696640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe45809bd80 0x7fe4580cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45d696640 1 -- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4580d2680 msgr2=0x7fe458157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45d696640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4580d2680 0x7fe458157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45d696640 1 -- 172.21.15.146:0/2566491676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4581e8010 con 0x7fe4581e7830 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe457fff640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4580d2680 0x7fe458157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45d696640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 0x7fe45806d110 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe450009fd0 tx=0x7fe45000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450014420 con 0x7fe4581e7830 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe450019070 con 0x7fe4581e7830 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4581e82f0 con 0x7fe4581e7830 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450014800 con 0x7fe4581e7830 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.418+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4581e88b0 con 0x7fe4581e7830 2024-04-25T15:20:50.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.419+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45806a0e0 con 0x7fe4581e7830 2024-04-25T15:20:50.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.420+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe4500149a0 con 0x7fe4581e7830 2024-04-25T15:20:50.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.420+0000 7fe4557fa640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4440744b0 0x7fe4440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.420+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fe450095e10 con 0x7fe4581e7830 2024-04-25T15:20:50.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.420+0000 7fe45ce95640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4440744b0 0x7fe4440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.421+0000 7fe45ce95640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4440744b0 0x7fe4440769c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe448034000 tx=0x7fe44802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.425+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4500601b0 con 0x7fe4581e7830 2024-04-25T15:20:50.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.602+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe45815cf90 con 0x7fe4581e7830 2024-04-25T15:20:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.602+0000 7fe4557fa640 1 -- 172.21.15.146:0/2566491676 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fe45005fab0 con 0x7fe4581e7830 2024-04-25T15:20:50.603 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4440744b0 msgr2=0x7fe4440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4440744b0 0x7fe4440769c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe448034000 tx=0x7fe44802e000 comp rx=0 tx=0).stop 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 msgr2=0x7fe45806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 0x7fe45806d110 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe450009fd0 tx=0x7fe45000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 shutdown_connections 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe4440744b0 0x7fe4440769c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4581e7830 0x7fe45806d110 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4580d2680 0x7fe458157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 --2- 172.21.15.146:0/2566491676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe45809bd80 0x7fe4580cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 >> 172.21.15.146:0/2566491676 conn(0x7fe4580b7400 msgr2=0x7fe458159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:50.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 shutdown_connections 2024-04-25T15:20:50.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.604+0000 7fe45f0f9640 1 -- 172.21.15.146:0/2566491676 wait complete. 2024-04-25T15:20:50.617 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:50.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.727+0000 7f9ac3772640 1 Processor -- start 2024-04-25T15:20:50.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.727+0000 7f9ac3772640 1 -- start start 2024-04-25T15:20:50.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac3772640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc0b18a0 0x7f9abc163cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac3772640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9abc164230 0x7f9abc0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac3772640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9abc0cf700 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac3772640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9abc0cfce0 con 0x7f9abc164230 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac3772640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9abc0d33b0 con 0x7f9abc0b18a0 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9abc164230 0x7f9abc0cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9abc164230 0x7f9abc0cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55422/0 (socket says 172.21.15.146:55422) 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/1980863884 learned_addr learned my addr 172.21.15.146:0/1980863884 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac150e640 1 --2- 172.21.15.146:0/1980863884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc0b18a0 0x7f9abc163cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/1980863884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc0b18a0 msgr2=0x7f9abc163cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 --2- 172.21.15.146:0/1980863884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc0b18a0 0x7f9abc163cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/1980863884 >> v1:172.21.15.177:6789/0 conn(0x7f9abc0ce2a0 legacy=0x7f9abc1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.729+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/1980863884 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc164640 con 0x7f9abc164230 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.730+0000 7f9ac0d0d640 1 --2- 172.21.15.146:0/1980863884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc164230 0x7f9abc0cdd60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009870 tx=0x7f9aac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c5b2ce6da7ccac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.730+0000 7f9ab3fff640 1 -- 172.21.15.146:0/1980863884 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac037070 con 0x7f9abc164230 2024-04-25T15:20:50.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.730+0000 7f9ab3fff640 1 -- 172.21.15.146:0/1980863884 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aac032440 con 0x7f9abc164230 2024-04-25T15:20:50.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.730+0000 7f9ab3fff640 1 -- 172.21.15.146:0/1980863884 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac032810 con 0x7f9abc164230 2024-04-25T15:20:50.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 -- 172.21.15.146:0/1980863884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc164230 msgr2=0x7f9abc0cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 --2- 172.21.15.146:0/1980863884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc164230 0x7f9abc0cdd60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009870 tx=0x7f9aac02a270 comp rx=0 tx=0).stop 2024-04-25T15:20:50.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 -- 172.21.15.146:0/1980863884 shutdown_connections 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 --2- 172.21.15.146:0/1980863884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc164230 0x7f9abc0cdd60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 --2- 172.21.15.146:0/1980863884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc0b18a0 0x7f9abc163cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 -- 172.21.15.146:0/1980863884 >> 172.21.15.146:0/1980863884 conn(0x7f9abc0b7430 msgr2=0x7f9abc0b7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 -- 172.21.15.146:0/1980863884 shutdown_connections 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.731+0000 7f9ac3772640 1 -- 172.21.15.146:0/1980863884 wait complete. 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.732+0000 7f9ac3772640 1 Processor -- start 2024-04-25T15:20:50.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.733+0000 7f9ac3772640 1 -- start start 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac3772640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9abc0b18a0 0x7f9abc1f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac3772640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 0x7f9abc1f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac3772640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc164230 0x7f9abc0b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac3772640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc0cf970 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac3772640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc0b3cd0 con 0x7f9abc0b18a0 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac3772640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9abc0b3fb0 con 0x7f9abc164230 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 0x7f9abc1f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 0x7f9abc1f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55426/0 (socket says 172.21.15.146:55426) 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/3829208758 learned_addr learned my addr 172.21.15.146:0/3829208758 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac150e640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9abc0b18a0 0x7f9abc1f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac1d0f640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc164230 0x7f9abc0b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc164230 msgr2=0x7f9abc0b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc164230 0x7f9abc0b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9abc0b18a0 msgr2=0x7f9abc1f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9abc0b18a0 0x7f9abc1f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 -- 172.21.15.146:0/3829208758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc0c6200 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac150e640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9abc0b18a0 0x7f9abc1f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac1d0f640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc164230 0x7f9abc0b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.734+0000 7f9ac0d0d640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 0x7f9abc1f6710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9aac02a2b0 tx=0x7f9aac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.735+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac041030 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.735+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aac030440 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.735+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9abc0c6470 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.735+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac037040 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.735+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9abc0c6a30 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.736+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9abc0b1f10 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.736+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9aac044070 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.737+0000 7f9ab1ffb640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9aa00744b0 0x7f9aa00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:50.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.737+0000 7f9ac150e640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9aa00744b0 0x7f9aa00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:50.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.737+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f9aac0b5d20 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.737+0000 7f9ac150e640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9aa00744b0 0x7f9aa00769c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9abc073a20 tx=0x7f9ab8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:50.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.741+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9abc164640 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.920+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9abc06a0e0 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.920+0000 7f9ab1ffb640 1 -- 172.21.15.146:0/3829208758 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f9aac0800c0 con 0x7f9abc0ce2a0 2024-04-25T15:20:50.921 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9aa00744b0 msgr2=0x7f9aa00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9aa00744b0 0x7f9aa00769c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9abc073a20 tx=0x7f9ab8008040 comp rx=0 tx=0).stop 2024-04-25T15:20:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 msgr2=0x7f9abc1f6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 0x7f9abc1f6710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9aac02a2b0 tx=0x7f9aac0046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 shutdown_connections 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9aa00744b0 0x7f9aa00769c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9abc164230 0x7f9abc0b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9abc0ce2a0 0x7f9abc1f6710 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 --2- 172.21.15.146:0/3829208758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9abc0b18a0 0x7f9abc1f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 >> 172.21.15.146:0/3829208758 conn(0x7f9abc0b7430 msgr2=0x7f9abc0cee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 shutdown_connections 2024-04-25T15:20:50.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:50.922+0000 7f9ac3772640 1 -- 172.21.15.146:0/3829208758 wait complete. 2024-04-25T15:20:50.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:51.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.044+0000 7fdb1fa30640 1 Processor -- start 2024-04-25T15:20:51.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.044+0000 7fdb1fa30640 1 -- start start 2024-04-25T15:20:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.045+0000 7fdb1fa30640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb180cbc90 0x7fdb18153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.045+0000 7fdb1fa30640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d0ac0 0x7fdb180d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.045+0000 7fdb1fa30640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb180d1410 con 0x7fdb180d01a0 2024-04-25T15:20:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.045+0000 7fdb1fa30640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdb181541e0 con 0x7fdb180cbc90 2024-04-25T15:20:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.045+0000 7fdb1fa30640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdb180cfd10 con 0x7fdb180d0ac0 2024-04-25T15:20:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb180cbc90 0x7fdb18153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb180cbc90 0x7fdb18153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55436/0 (socket says 172.21.15.146:55436) 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/254515521 learned_addr learned my addr 172.21.15.146:0/254515521 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1dfcd640 1 --2- 172.21.15.146:0/254515521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d0ac0 0x7fdb180d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/254515521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d0ac0 msgr2=0x7fdb180d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 --2- 172.21.15.146:0/254515521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d0ac0 0x7fdb180d0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/254515521 >> v1:172.21.15.177:6789/0 conn(0x7fdb180d01a0 legacy=0x7fdb1815dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.046+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/254515521 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb180cf420 con 0x7fdb180cbc90 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1d7cc640 1 --2- 172.21.15.146:0/254515521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18153ca0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400bb10 tx=0x7fdb1402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9841fb30469ed53e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb0ffff640 1 -- 172.21.15.146:0/254515521 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14036020 con 0x7fdb180cbc90 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb0ffff640 1 -- 172.21.15.146:0/254515521 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb14032820 con 0x7fdb180cbc90 2024-04-25T15:20:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb0ffff640 1 -- 172.21.15.146:0/254515521 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14032bf0 con 0x7fdb180cbc90 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1fa30640 1 -- 172.21.15.146:0/254515521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 msgr2=0x7fdb18153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/254515521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18153ca0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400bb10 tx=0x7fdb1402c920 comp rx=0 tx=0).stop 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1fa30640 1 -- 172.21.15.146:0/254515521 shutdown_connections 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/254515521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d0ac0 0x7fdb180d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/254515521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18153ca0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.047+0000 7fdb1fa30640 1 -- 172.21.15.146:0/254515521 >> 172.21.15.146:0/254515521 conn(0x7fdb180ce0c0 msgr2=0x7fdb180ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.048+0000 7fdb1fa30640 1 -- 172.21.15.146:0/254515521 shutdown_connections 2024-04-25T15:20:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.048+0000 7fdb1fa30640 1 -- 172.21.15.146:0/254515521 wait complete. 2024-04-25T15:20:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.049+0000 7fdb1fa30640 1 Processor -- start 2024-04-25T15:20:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.049+0000 7fdb1fa30640 1 -- start start 2024-04-25T15:20:51.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d01a0 0x7fdb1815a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d0ac0 0x7fdb181f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180cf9a0 con 0x7fdb180cbc90 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180cf6c0 con 0x7fdb180d0ac0 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdb180d05b0 con 0x7fdb180d01a0 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55452/0 (socket says 172.21.15.146:55452) 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/3635235683 learned_addr learned my addr 172.21.15.146:0/3635235683 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1cfcb640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d01a0 0x7fdb1815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1dfcd640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d0ac0 0x7fdb181f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d01a0 msgr2=0x7fdb1815a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d01a0 0x7fdb1815a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d0ac0 msgr2=0x7fdb181f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d0ac0 0x7fdb181f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 -- 172.21.15.146:0/3635235683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb181f3350 con 0x7fdb180cbc90 2024-04-25T15:20:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1d7cc640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18158590 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400b2c0 tx=0x7fdb140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14036040 con 0x7fdb180cbc90 2024-04-25T15:20:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb1403bd60 con 0x7fdb180cbc90 2024-04-25T15:20:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb18157ab0 con 0x7fdb180cbc90 2024-04-25T15:20:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.051+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14045630 con 0x7fdb180cbc90 2024-04-25T15:20:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.052+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb18158070 con 0x7fdb180cbc90 2024-04-25T15:20:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.052+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb1806a0e0 con 0x7fdb180cbc90 2024-04-25T15:20:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.053+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdb1404c060 con 0x7fdb180cbc90 2024-04-25T15:20:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.054+0000 7fdb0dffb640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdafc0744b0 0x7fdafc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.054+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fdb140b8610 con 0x7fdb180cbc90 2024-04-25T15:20:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.054+0000 7fdb1cfcb640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdafc0744b0 0x7fdafc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.054+0000 7fdb1cfcb640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdafc0744b0 0x7fdafc0769c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdb180b4da0 tx=0x7fdb08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.058+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb14040070 con 0x7fdb180cbc90 2024-04-25T15:20:51.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.245+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdb180711c0 con 0x7fdb180cbc90 2024-04-25T15:20:51.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.246+0000 7fdb0dffb640 1 -- 172.21.15.146:0/3635235683 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fdb1402f070 con 0x7fdb180cbc90 2024-04-25T15:20:51.250 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdafc0744b0 msgr2=0x7fdafc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdafc0744b0 0x7fdafc0769c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdb180b4da0 tx=0x7fdb08009040 comp rx=0 tx=0).stop 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 msgr2=0x7fdb18158590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18158590 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400b2c0 tx=0x7fdb140046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 shutdown_connections 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdafc0744b0 0x7fdafc0769c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d0ac0 0x7fdb181f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb180d01a0 0x7fdb1815a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 --2- 172.21.15.146:0/3635235683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180cbc90 0x7fdb18158590 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 >> 172.21.15.146:0/3635235683 conn(0x7fdb180ce0c0 msgr2=0x7fdb181531b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:51.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 shutdown_connections 2024-04-25T15:20:51.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.248+0000 7fdb1fa30640 1 -- 172.21.15.146:0/3635235683 wait complete. 2024-04-25T15:20:51.260 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:20:51.260 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:20:51.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:51.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.388+0000 7fade170d640 1 Processor -- start 2024-04-25T15:20:51.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.388+0000 7fade170d640 1 -- start start 2024-04-25T15:20:51.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fade170d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 0x7faddc15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fade170d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faddc06f6d0 0x7faddc0b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fade170d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faddc09c2a0 con 0x7faddc06f2c0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fade170d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7faddc0d1d60 con 0x7faddc06f6d0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fade170d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faddc09bce0 con 0x7faddc0d21f0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fadda7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7faddc06f2c0 0x7faddc0b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45868/0 (socket says 172.21.15.146:45868) 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7fadda7fc640 1 -- 172.21.15.146:0/3043845537 learned_addr learned my addr 172.21.15.146:0/3043845537 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7faddb7fe640 1 --2- 172.21.15.146:0/3043845537 >> v2:172.21.15.177:3300/0 conn(0x7faddc06f6d0 0x7faddc0b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.389+0000 7faddaffd640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 0x7faddc15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997364685 0 0) 0x7faddc09c2a0 con 0x7faddc06f2c0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadc8003520 con 0x7faddc06f2c0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1903921316 0 0) 0x7fadc8003520 con 0x7faddc06f2c0 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 msgr2=0x7faddc15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 0x7faddc15ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 >> v2:172.21.15.177:3300/0 conn(0x7faddc06f6d0 msgr2=0x7faddc0b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 --2- 172.21.15.146:0/3043845537 >> v2:172.21.15.177:3300/0 conn(0x7faddc06f6d0 0x7faddc0b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc09bfc0 con 0x7faddc06f2c0 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fadc4001670 con 0x7faddc06f2c0 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 >> v1:172.21.15.146:6789/0 conn(0x7faddc06f2c0 legacy=0x7faddc0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadc80039b0 0x7fadc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fadc8004800 0x7fadc8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 0x7fadc8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faddc0d1d60 con 0x7fadc8004800 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faddc09bce0 con 0x7fadc80039b0 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fadc800ba40 con 0x7faddc0d21f0 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadda7fc640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fadc8004800 0x7fadc8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7faddaffd640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadc80039b0 0x7fadc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7faddb7fe640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 0x7fadc8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadda7fc640 1 -- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 msgr2=0x7fadc8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadda7fc640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faddc0d21f0 0x7fadc8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadda7fc640 1 -- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadc80039b0 msgr2=0x7fadc8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadda7fc640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadc80039b0 0x7fadc8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7fadda7fc640 1 -- 172.21.15.146:0/3043845537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc8003520 con 0x7fadc8004800 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.390+0000 7faddaffd640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadc80039b0 0x7fadc8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.391+0000 7fadda7fc640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fadc8004800 0x7fadc8004c10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5e95f8041800b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.391+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4007d50 con 0x7fadc8004800 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.391+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc4005ad0 con 0x7fadc8004800 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.391+0000 7fadd9ffb640 1 -- 172.21.15.146:0/3043845537 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4007070 con 0x7fadc8004800 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7fade170d640 1 -- 172.21.15.146:0/3043845537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fadc8004800 msgr2=0x7fadc8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7fade170d640 1 --2- 172.21.15.146:0/3043845537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fadc8004800 0x7fadc8004c10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc8005830 comp rx=0 tx=0).stop 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7faddb7fe640 1 -- 172.21.15.146:0/3043845537 reap_dead start 2024-04-25T15:20:51.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7fade170d640 1 -- 172.21.15.146:0/3043845537 shutdown_connections 2024-04-25T15:20:51.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7fade170d640 1 -- 172.21.15.146:0/3043845537 >> 172.21.15.146:0/3043845537 conn(0x7faddc0b7530 msgr2=0x7faddc0b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:51.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7fade170d640 1 -- 172.21.15.146:0/3043845537 shutdown_connections 2024-04-25T15:20:51.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.392+0000 7fade170d640 1 -- 172.21.15.146:0/3043845537 wait complete. 2024-04-25T15:20:51.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.394+0000 7fade170d640 1 Processor -- start 2024-04-25T15:20:51.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.394+0000 7fade170d640 1 -- start start 2024-04-25T15:20:51.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fade170d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadc8004800 0x7faddc06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fade170d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 0x7faddc154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fade170d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faddc14d2f0 0x7faddc1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fade170d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faddc09b580 con 0x7faddc14cee0 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fade170d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faddc09b400 con 0x7faddc14d2f0 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fade170d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faddc0b3de0 con 0x7fadc8004800 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 0x7faddc154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 0x7faddc154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55458/0 (socket says 172.21.15.146:55458) 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 -- 172.21.15.146:0/778933341 learned_addr learned my addr 172.21.15.146:0/778933341 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7faddb7fe640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faddc14d2f0 0x7faddc1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7faddaffd640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadc8004800 0x7faddc06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 -- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadc8004800 msgr2=0x7faddc06ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadc8004800 0x7faddc06ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 -- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faddc14d2f0 msgr2=0x7faddc1550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faddc14d2f0 0x7faddc1550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7fadda7fc640 1 -- 172.21.15.146:0/778933341 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc06dbb0 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.395+0000 7faddb7fe640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faddc14d2f0 0x7faddc1550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.396+0000 7fadda7fc640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 0x7faddc154bb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c300 tx=0x7faddc0d1cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.396+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4042070 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.396+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc4007480 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.396+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4001cb0 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.396+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc06de90 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.396+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc14d9d0 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.397+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faddc06a0e0 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.398+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fadc4045070 con 0x7faddc14cee0 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.398+0000 7fadd37fe640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadbc0744b0 0x7fadbc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.398+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fadc40b6300 con 0x7faddc14cee0 2024-04-25T15:20:51.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.398+0000 7faddaffd640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadbc0744b0 0x7fadbc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.399+0000 7faddaffd640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadbc0744b0 0x7fadbc0769c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009670 tx=0x7fadcc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.403+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadc404c030 con 0x7faddc14cee0 2024-04-25T15:20:51.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.581+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faddc0b2f10 con 0x7faddc14cee0 2024-04-25T15:20:51.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.582+0000 7fadd37fe640 1 -- 172.21.15.146:0/778933341 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7fadc4080770 con 0x7faddc14cee0 2024-04-25T15:20:51.583 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadbc0744b0 msgr2=0x7fadbc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadbc0744b0 0x7fadbc0769c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009670 tx=0x7fadcc00b040 comp rx=0 tx=0).stop 2024-04-25T15:20:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 msgr2=0x7faddc154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 0x7faddc154bb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c300 tx=0x7faddc0d1cd0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 shutdown_connections 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fadbc0744b0 0x7fadbc0769c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faddc14d2f0 0x7faddc1550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faddc14cee0 0x7faddc154bb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 --2- 172.21.15.146:0/778933341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadc8004800 0x7faddc06ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 >> 172.21.15.146:0/778933341 conn(0x7faddc0b7530 msgr2=0x7faddc0b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 shutdown_connections 2024-04-25T15:20:51.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.584+0000 7fade170d640 1 -- 172.21.15.146:0/778933341 wait complete. 2024-04-25T15:20:51.597 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:51.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:51.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.707+0000 7f1ca7796640 1 Processor -- start 2024-04-25T15:20:51.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.707+0000 7f1ca7796640 1 -- start start 2024-04-25T15:20:51.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca7796640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1ca009bba0 0x7f1ca00b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca7796640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca00b1960 con 0x7f1ca00d2d90 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca7796640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca00b3ce0 con 0x7f1ca00ccd40 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca7796640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca00b1c00 con 0x7f1ca009bba0 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca4d31640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1ca00d2d90 0x7f1ca00b2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45872/0 (socket says 172.21.15.146:45872) 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca4d31640 1 -- 172.21.15.146:0/666800680 learned_addr learned my addr 172.21.15.146:0/666800680 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca5d33640 1 --2- 172.21.15.146:0/666800680 >> v2:172.21.15.177:3300/0 conn(0x7f1ca009bba0 0x7f1ca00b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507789446 0 0) 0x7f1ca00b1960 con 0x7f1ca00d2d90 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c8c003520 con 0x7f1ca00d2d90 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1296121464 0 0) 0x7f1c8c003520 con 0x7f1ca00d2d90 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 >> v2:172.21.15.177:3300/0 conn(0x7f1ca009bba0 msgr2=0x7f1ca00b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 --2- 172.21.15.146:0/666800680 >> v2:172.21.15.177:3300/0 conn(0x7f1ca009bba0 0x7f1ca00b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 >> v1:172.21.15.177:6789/0 conn(0x7f1ca00ccd40 legacy=0x7f1ca00d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca00b1ee0 con 0x7f1ca00d2d90 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f1c90001690 con 0x7f1ca00d2d90 2024-04-25T15:20:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.709+0000 7f1ca5d33640 1 --2- 172.21.15.146:0/666800680 >> v2:172.21.15.177:3300/0 conn(0x7f1ca009bba0 0x7f1ca00b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 >> v1:172.21.15.146:6789/0 conn(0x7f1ca00d2d90 legacy=0x7f1ca00b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c8c0039b0 0x7f1c8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c004800 0x7f1c8c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0051c0 0x7f1c8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca00b1c00 con 0x7f1c8c0051c0 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c00c160 con 0x7f1c8c004800 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c00c360 con 0x7f1c8c0039b0 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5d33640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0051c0 0x7f1c8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca4d31640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c004800 0x7f1c8c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5532640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c8c0039b0 0x7f1c8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5d33640 1 -- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c8c0039b0 msgr2=0x7f1c8c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5d33640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c8c0039b0 0x7f1c8c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5d33640 1 -- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c004800 msgr2=0x7f1c8c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5d33640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c004800 0x7f1c8c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.710+0000 7f1ca5d33640 1 -- 172.21.15.146:0/666800680 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c003520 con 0x7f1c8c0051c0 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1ca5d33640 1 -- 172.21.15.146:0/666800680 reap_dead start 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1ca5d33640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0051c0 0x7f1c8c00b9c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009c90 tx=0x7f1c8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4977e739982e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98009860 con 0x7f1c8c0051c0 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c98004020 con 0x7f1c8c0051c0 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1c97fff640 1 -- 172.21.15.146:0/666800680 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98004410 con 0x7f1c8c0051c0 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1ca7796640 1 -- 172.21.15.146:0/666800680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0051c0 msgr2=0x7f1c8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.711+0000 7f1ca7796640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0051c0 0x7f1c8c00b9c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009c90 tx=0x7f1c8c005830 comp rx=0 tx=0).stop 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.712+0000 7f1ca7796640 1 -- 172.21.15.146:0/666800680 shutdown_connections 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.712+0000 7f1ca7796640 1 --2- 172.21.15.146:0/666800680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0051c0 0x7f1c8c00b9c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.712+0000 7f1ca7796640 1 -- 172.21.15.146:0/666800680 >> 172.21.15.146:0/666800680 conn(0x7f1ca00b7400 msgr2=0x7f1ca00b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.712+0000 7f1ca7796640 1 -- 172.21.15.146:0/666800680 shutdown_connections 2024-04-25T15:20:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.712+0000 7f1ca7796640 1 -- 172.21.15.146:0/666800680 wait complete. 2024-04-25T15:20:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.713+0000 7f1ca7796640 1 Processor -- start 2024-04-25T15:20:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.713+0000 7f1ca7796640 1 -- start start 2024-04-25T15:20:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c0051c0 0x7f1ca01620f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 0x7f1ca006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ca0155e10 0x7f1ca006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca00b3940 con 0x7f1ca0155a00 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca009bfb0 con 0x7f1c8c0051c0 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca00b3ac0 con 0x7f1ca0155e10 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 0x7f1ca006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 0x7f1ca006e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55476/0 (socket says 172.21.15.146:55476) 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 -- 172.21.15.146:0/985233761 learned_addr learned my addr 172.21.15.146:0/985233761 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 -- 172.21.15.146:0/985233761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ca0155e10 msgr2=0x7f1ca006ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ca0155e10 0x7f1ca006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 -- 172.21.15.146:0/985233761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c0051c0 msgr2=0x7f1ca01620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c0051c0 0x7f1ca01620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 -- 172.21.15.146:0/985233761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca0162630 con 0x7f1ca0155a00 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca4d31640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 0x7f1ca006e920 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1c90000c00 tx=0x7f1c90007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000f7b0 con 0x7f1ca0155a00 2024-04-25T15:20:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c90018070 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca00cba90 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.715+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000fbb0 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.716+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca00cc050 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.717+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca009b440 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.717+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1c9000fd50 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.717+0000 7f1c95ffb640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c840744e0 0x7f1c840769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.717+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f1c900954f0 con 0x7f1ca0155a00 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.717+0000 7f1ca5532640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c840744e0 0x7f1c840769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.718+0000 7f1ca5532640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c840744e0 0x7f1c840769f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c000f80 tx=0x7f1c9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:51.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.722+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ca009b440 con 0x7f1ca0155a00 2024-04-25T15:20:51.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.895+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ca006a0e0 con 0x7f1ca0155a00 2024-04-25T15:20:51.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.896+0000 7f1c95ffb640 1 -- 172.21.15.146:0/985233761 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f1ca0162630 con 0x7f1ca0155a00 2024-04-25T15:20:51.897 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:51.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.897+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c840744e0 msgr2=0x7f1c840769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.897+0000 7f1ca7796640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c840744e0 0x7f1c840769f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c000f80 tx=0x7f1c9c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:51.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 msgr2=0x7f1ca006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:51.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 0x7f1ca006e920 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1c90000c00 tx=0x7f1c90007430 comp rx=0 tx=0).stop 2024-04-25T15:20:51.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 shutdown_connections 2024-04-25T15:20:51.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c840744e0 0x7f1c840769f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ca0155e10 0x7f1ca006ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ca0155a00 0x7f1ca006e920 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 --2- 172.21.15.146:0/985233761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c8c0051c0 0x7f1ca01620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:51.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 >> 172.21.15.146:0/985233761 conn(0x7f1ca00b7400 msgr2=0x7f1ca0070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:51.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 shutdown_connections 2024-04-25T15:20:51.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:51.898+0000 7f1ca7796640 1 -- 172.21.15.146:0/985233761 wait complete. 2024-04-25T15:20:51.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:52.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.021+0000 7fb434bce640 1 Processor -- start 2024-04-25T15:20:52.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.021+0000 7fb434bce640 1 -- start start 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.022+0000 7fb434bce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300b2cd0 0x7fb4300d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.022+0000 7fb434bce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb430071d30 0x7fb4300cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.022+0000 7fb434bce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4300d2260 con 0x7fb4300d3300 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.022+0000 7fb434bce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb4300d2540 con 0x7fb430071d30 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.022+0000 7fb434bce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb4300d2820 con 0x7fb4300b2cd0 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb430071d30 0x7fb4300cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb430071d30 0x7fb4300cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55488/0 (socket says 172.21.15.146:55488) 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 -- 172.21.15.146:0/1056504060 learned_addr learned my addr 172.21.15.146:0/1056504060 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42e59c640 1 --2- 172.21.15.146:0/1056504060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300b2cd0 0x7fb4300d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 -- 172.21.15.146:0/1056504060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300b2cd0 msgr2=0x7fb4300d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:52.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 --2- 172.21.15.146:0/1056504060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300b2cd0 0x7fb4300d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 -- 172.21.15.146:0/1056504060 >> v1:172.21.15.177:6789/0 conn(0x7fb4300d3300 legacy=0x7fb430071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.023+0000 7fb42ed9d640 1 -- 172.21.15.146:0/1056504060 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4300d2b00 con 0x7fb430071d30 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.024+0000 7fb42ed9d640 1 --2- 172.21.15.146:0/1056504060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb430071d30 0x7fb4300cde40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb428009870 tx=0x7fb42802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2ba0046b81f2c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.024+0000 7fb42d59a640 1 -- 172.21.15.146:0/1056504060 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428037070 con 0x7fb430071d30 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.024+0000 7fb42d59a640 1 -- 172.21.15.146:0/1056504060 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb428032440 con 0x7fb430071d30 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.024+0000 7fb42d59a640 1 -- 172.21.15.146:0/1056504060 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428032810 con 0x7fb430071d30 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.024+0000 7fb434bce640 1 -- 172.21.15.146:0/1056504060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb430071d30 msgr2=0x7fb4300cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.024+0000 7fb434bce640 1 --2- 172.21.15.146:0/1056504060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb430071d30 0x7fb4300cde40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb428009870 tx=0x7fb42802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.025+0000 7fb434bce640 1 -- 172.21.15.146:0/1056504060 shutdown_connections 2024-04-25T15:20:52.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.025+0000 7fb434bce640 1 --2- 172.21.15.146:0/1056504060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb430071d30 0x7fb4300cde40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.025+0000 7fb434bce640 1 --2- 172.21.15.146:0/1056504060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300b2cd0 0x7fb4300d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.025+0000 7fb434bce640 1 -- 172.21.15.146:0/1056504060 >> 172.21.15.146:0/1056504060 conn(0x7fb4300b52e0 msgr2=0x7fb4300b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.025+0000 7fb434bce640 1 -- 172.21.15.146:0/1056504060 shutdown_connections 2024-04-25T15:20:52.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.025+0000 7fb434bce640 1 -- 172.21.15.146:0/1056504060 wait complete. 2024-04-25T15:20:52.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.026+0000 7fb434bce640 1 Processor -- start 2024-04-25T15:20:52.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.027+0000 7fb434bce640 1 -- start start 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb434bce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 0x7fb43015d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb434bce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300d3300 0x7fb4300b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb434bce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4300b7b80 0x7fb4300b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb434bce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb43008dbf0 con 0x7fb4300b2cd0 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb434bce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb4300cfb70 con 0x7fb4300b7b80 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb434bce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb43008dd70 con 0x7fb4300d3300 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 0x7fb43015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 0x7fb43015d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55490/0 (socket says 172.21.15.146:55490) 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 -- 172.21.15.146:0/1838703033 learned_addr learned my addr 172.21.15.146:0/1838703033 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42dd9b640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300d3300 0x7fb4300b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42ed9d640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4300b7b80 0x7fb4300b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 -- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300d3300 msgr2=0x7fb4300b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300d3300 0x7fb4300b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 -- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4300b7b80 msgr2=0x7fb4300b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4300b7b80 0x7fb4300b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 -- 172.21.15.146:0/1838703033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43015dc90 con 0x7fb4300b2cd0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb42e59c640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 0x7fb43015d750 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb42000efc0 tx=0x7fb42000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.028+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420018020 con 0x7fb4300b2cd0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.029+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb420014480 con 0x7fb4300b2cd0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.029+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4300b1be0 con 0x7fb4300b2cd0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.029+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420014850 con 0x7fb4300b2cd0 2024-04-25T15:20:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.029+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4300b21a0 con 0x7fb4300b2cd0 2024-04-25T15:20:52.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.030+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43006a0e0 con 0x7fb4300b2cd0 2024-04-25T15:20:52.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.030+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb42001b020 con 0x7fb4300b2cd0 2024-04-25T15:20:52.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.031+0000 7fb41effd640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4100744b0 0x7fb4100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.031+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fb4200964d0 con 0x7fb4300b2cd0 2024-04-25T15:20:52.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.031+0000 7fb42dd9b640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4100744b0 0x7fb4100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.031+0000 7fb42dd9b640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4100744b0 0x7fb4100769c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb43008d8e0 tx=0x7fb424009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.035+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb420022020 con 0x7fb4300b2cd0 2024-04-25T15:20:52.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.211+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb4300b40e0 con 0x7fb4300b2cd0 2024-04-25T15:20:52.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.211+0000 7fb41effd640 1 -- 172.21.15.146:0/1838703033 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fb420010040 con 0x7fb4300b2cd0 2024-04-25T15:20:52.213 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4100744b0 msgr2=0x7fb4100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4100744b0 0x7fb4100769c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb43008d8e0 tx=0x7fb424009040 comp rx=0 tx=0).stop 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 msgr2=0x7fb43015d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 0x7fb43015d750 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb42000efc0 tx=0x7fb42000c490 comp rx=0 tx=0).stop 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 shutdown_connections 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4100744b0 0x7fb4100769c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4300b7b80 0x7fb4300b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4300d3300 0x7fb4300b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 --2- 172.21.15.146:0/1838703033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4300b2cd0 0x7fb43015d750 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 >> 172.21.15.146:0/1838703033 conn(0x7fb4300b52e0 msgr2=0x7fb43015faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 shutdown_connections 2024-04-25T15:20:52.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.213+0000 7fb434bce640 1 -- 172.21.15.146:0/1838703033 wait complete. 2024-04-25T15:20:52.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.335+0000 7fa3d26d1640 1 Processor -- start 2024-04-25T15:20:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.336+0000 7fa3d26d1640 1 -- start start 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d26d1640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa3cc06d770 0x7fa3cc15de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d26d1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3cc06c550 0x7fa3cc06c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d26d1640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d26d1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa3cc0b1e70 con 0x7fa3cc06c550 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d26d1640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa3cc09b620 con 0x7fa3cc06d770 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d26d1640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc0b1540 con 0x7fa3cc06cea0 2024-04-25T15:20:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3cc06c550 0x7fa3cc06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3cc06c550 0x7fa3cc06c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55492/0 (socket says 172.21.15.146:55492) 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 -- 172.21.15.146:0/3883854829 learned_addr learned my addr 172.21.15.146:0/3883854829 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 -- 172.21.15.146:0/3883854829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 msgr2=0x7fa3cc161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cbfff640 1 --2- 172.21.15.146:0/3883854829 >> v2:172.21.15.177:3300/0 conn(0x7fa3cc06d770 0x7fa3cc15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3d0c6e640 1 --2- 172.21.15.146:0/3883854829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 --2- 172.21.15.146:0/3883854829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 -- 172.21.15.146:0/3883854829 >> v2:172.21.15.177:3300/0 conn(0x7fa3cc06d770 msgr2=0x7fa3cc15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 --2- 172.21.15.146:0/3883854829 >> v2:172.21.15.177:3300/0 conn(0x7fa3cc06d770 0x7fa3cc15de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.337+0000 7fa3cb7fe640 1 -- 172.21.15.146:0/3883854829 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc0b1820 con 0x7fa3cc06c550 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.338+0000 7fa3cb7fe640 1 --2- 172.21.15.146:0/3883854829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06c550 0x7fa3cc06c960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0b1750 tx=0x7fa3c002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fcd06defbde792 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.338+0000 7fa3caffd640 1 -- 172.21.15.146:0/3883854829 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0037070 con 0x7fa3cc06c550 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.338+0000 7fa3caffd640 1 -- 172.21.15.146:0/3883854829 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c0032440 con 0x7fa3cc06c550 2024-04-25T15:20:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.338+0000 7fa3caffd640 1 -- 172.21.15.146:0/3883854829 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0032810 con 0x7fa3cc06c550 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 -- 172.21.15.146:0/3883854829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06c550 msgr2=0x7fa3cc06c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/3883854829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06c550 0x7fa3cc06c960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0b1750 tx=0x7fa3c002a130 comp rx=0 tx=0).stop 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 -- 172.21.15.146:0/3883854829 shutdown_connections 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/3883854829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/3883854829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06c550 0x7fa3cc06c960 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/3883854829 >> v2:172.21.15.177:3300/0 conn(0x7fa3cc06d770 0x7fa3cc15de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 -- 172.21.15.146:0/3883854829 >> 172.21.15.146:0/3883854829 conn(0x7fa3cc073a30 msgr2=0x7fa3cc0b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 -- 172.21.15.146:0/3883854829 shutdown_connections 2024-04-25T15:20:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.339+0000 7fa3d26d1640 1 -- 172.21.15.146:0/3883854829 wait complete. 2024-04-25T15:20:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.341+0000 7fa3d26d1640 1 Processor -- start 2024-04-25T15:20:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.341+0000 7fa3d26d1640 1 -- start start 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.342+0000 7fa3d26d1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3cc06c550 0x7fa3cc09ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.342+0000 7fa3d26d1640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc14a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.342+0000 7fa3d26d1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 0x7fa3cc15f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.342+0000 7fa3d26d1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc06d2b0 con 0x7fa3cc06d770 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.342+0000 7fa3d26d1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc15e390 con 0x7fa3cc06c550 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.342+0000 7fa3d26d1640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc09b2b0 con 0x7fa3cc06cea0 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 0x7fa3cc15f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 0x7fa3cc15f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55496/0 (socket says 172.21.15.146:55496) 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 -- 172.21.15.146:0/2927097612 learned_addr learned my addr 172.21.15.146:0/2927097612 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3cbfff640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3cc06c550 0x7fa3cc09ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3cb7fe640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc14a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 -- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 msgr2=0x7fa3cc14a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc14a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 -- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3cc06c550 msgr2=0x7fa3cc09ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3cc06c550 0x7fa3cc09ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 -- 172.21.15.146:0/2927097612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc1613f0 con 0x7fa3cc06d770 2024-04-25T15:20:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3cb7fe640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc14a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d0c6e640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 0x7fa3cc15f800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400cb80 tx=0x7fa3c400a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4015070 con 0x7fa3cc06d770 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c400abd0 con 0x7fa3cc06d770 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc0afc00 con 0x7fa3cc06d770 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400d040 con 0x7fa3cc06d770 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.343+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3cc0b0100 con 0x7fa3cc06d770 2024-04-25T15:20:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.344+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3cc0afe70 con 0x7fa3cc06d770 2024-04-25T15:20:52.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.345+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa3c4018070 con 0x7fa3cc06d770 2024-04-25T15:20:52.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.345+0000 7fa3c8ff9640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3b80744e0 0x7fa3b80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.345+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fa3c4095510 con 0x7fa3cc06d770 2024-04-25T15:20:52.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.345+0000 7fa3cbfff640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3b80744e0 0x7fa3b80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.346+0000 7fa3cbfff640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3b80744e0 0x7fa3b80769f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc006fd0 tx=0x7fa3bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.350+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3cc1613f0 con 0x7fa3cc06d770 2024-04-25T15:20:52.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.524+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa3cc15c4f0 con 0x7fa3cc06d770 2024-04-25T15:20:52.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.525+0000 7fa3c8ff9640 1 -- 172.21.15.146:0/2927097612 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fa3c405f940 con 0x7fa3cc06d770 2024-04-25T15:20:52.525 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3b80744e0 msgr2=0x7fa3b80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3b80744e0 0x7fa3b80769f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc006fd0 tx=0x7fa3bc008040 comp rx=0 tx=0).stop 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 msgr2=0x7fa3cc15f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 0x7fa3cc15f800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400cb80 tx=0x7fa3c400a4c0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 shutdown_connections 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa3b80744e0 0x7fa3b80769f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3cc06d770 0x7fa3cc15f800 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3cc06cea0 0x7fa3cc14a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 --2- 172.21.15.146:0/2927097612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3cc06c550 0x7fa3cc09ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 >> 172.21.15.146:0/2927097612 conn(0x7fa3cc073a30 msgr2=0x7fa3cc15bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 shutdown_connections 2024-04-25T15:20:52.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.527+0000 7fa3d26d1640 1 -- 172.21.15.146:0/2927097612 wait complete. 2024-04-25T15:20:52.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:52.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.648+0000 7f9f42f55640 1 Processor -- start 2024-04-25T15:20:52.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.648+0000 7f9f42f55640 1 -- start start 2024-04-25T15:20:52.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f42f55640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9f3c071d30 0x7f9f3c0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f42f55640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f3c0d2260 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f42f55640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f3c0d2540 con 0x7f9f3c0d3300 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f42f55640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f9f3c0d2820 con 0x7f9f3c071d30 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f40cf1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9f3c0b2cd0 0x7f9f3c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45882/0 (socket says 172.21.15.146:45882) 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f40cf1640 1 -- 172.21.15.146:0/2734644154 learned_addr learned my addr 172.21.15.146:0/2734644154 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f3bfff640 1 --2- 172.21.15.146:0/2734644154 >> v2:172.21.15.177:3300/0 conn(0x7f9f3c071d30 0x7f9f3c0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f3b7fe640 1 -- 172.21.15.146:0/2734644154 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2874472322 0 0) 0x7f9f3c0d2260 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.649+0000 7f9f3b7fe640 1 -- 172.21.15.146:0/2734644154 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f24003520 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.650+0000 7f9f3bfff640 1 -- 172.21.15.146:0/2734644154 >> v1:172.21.15.177:6789/0 conn(0x7f9f3c0d3300 legacy=0x7f9f3c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.650+0000 7f9f3b7fe640 1 -- 172.21.15.146:0/2734644154 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1255377229 0 0) 0x7f9f24003520 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.650+0000 7f9f3bfff640 1 -- 172.21.15.146:0/2734644154 >> v1:172.21.15.146:6789/0 conn(0x7f9f3c0b2cd0 legacy=0x7f9f3c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.650+0000 7f9f3bfff640 1 -- 172.21.15.146:0/2734644154 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3c0d2b00 con 0x7f9f3c071d30 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.650+0000 7f9f3bfff640 1 --2- 172.21.15.146:0/2734644154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c071d30 0x7f9f3c0d1c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9f300097a0 tx=0x7f9f3002a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6672ff0af5c20b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f3b7fe640 1 -- 172.21.15.146:0/2734644154 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f30037070 con 0x7f9f3c071d30 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f3b7fe640 1 -- 172.21.15.146:0/2734644154 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3002aad0 con 0x7f9f3c071d30 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f3b7fe640 1 -- 172.21.15.146:0/2734644154 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0017f0 con 0x7f9f3c071d30 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 -- 172.21.15.146:0/2734644154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c071d30 msgr2=0x7f9f3c0d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 --2- 172.21.15.146:0/2734644154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c071d30 0x7f9f3c0d1c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9f300097a0 tx=0x7f9f3002a040 comp rx=0 tx=0).stop 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 -- 172.21.15.146:0/2734644154 shutdown_connections 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 --2- 172.21.15.146:0/2734644154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c071d30 0x7f9f3c0d1c80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 -- 172.21.15.146:0/2734644154 >> 172.21.15.146:0/2734644154 conn(0x7f9f3c0b52e0 msgr2=0x7f9f3c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 -- 172.21.15.146:0/2734644154 shutdown_connections 2024-04-25T15:20:52.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.651+0000 7f9f42f55640 1 -- 172.21.15.146:0/2734644154 wait complete. 2024-04-25T15:20:52.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.653+0000 7f9f42f55640 1 Processor -- start 2024-04-25T15:20:52.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.653+0000 7f9f42f55640 1 -- start start 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.654+0000 7f9f42f55640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 0x7f9f3c1e85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.654+0000 7f9f42f55640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f3c0d3300 0x7f9f3c1e95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.654+0000 7f9f42f55640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c158060 0x7f9f3c0cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.654+0000 7f9f42f55640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3c08dbf0 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.654+0000 7f9f42f55640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3c0cfb70 con 0x7f9f3c158060 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.654+0000 7f9f42f55640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3c08dd70 con 0x7f9f3c0d3300 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 0x7f9f3c1e85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 0x7f9f3c1e85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55510/0 (socket says 172.21.15.146:55510) 2024-04-25T15:20:52.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 -- 172.21.15.146:0/1584529894 learned_addr learned my addr 172.21.15.146:0/1584529894 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f3bfff640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f3c0d3300 0x7f9f3c1e95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f414f2640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c158060 0x7f9f3c0cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 -- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f3c0d3300 msgr2=0x7f9f3c1e95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f3c0d3300 0x7f9f3c1e95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 -- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c158060 msgr2=0x7f9f3c0cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c158060 0x7f9f3c0cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 -- 172.21.15.146:0/1584529894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3c1e9fa0 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f40cf1640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 0x7f9f3c1e85e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00c4e0 tx=0x7f9f2c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c017070 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2c007480 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3c1e7820 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0059a0 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.655+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3c1e7de0 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.656+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f3c06a0e0 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.657+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9f2c01a070 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.657+0000 7f9f397fa640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f280744e0 0x7f9f280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.658+0000 7f9f3bfff640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f280744e0 0x7f9f280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.658+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f9f2c0949b0 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.658+0000 7f9f3bfff640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f280744e0 0x7f9f280769f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9f30034000 tx=0x7f9f3002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.662+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f2c05e680 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.838+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9f3c0d0e20 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.839+0000 7f9f397fa640 1 -- 172.21.15.146:0/1584529894 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f9f2c062980 con 0x7f9f3c0b2cd0 2024-04-25T15:20:52.840 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:52.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f280744e0 msgr2=0x7f9f280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f280744e0 0x7f9f280769f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9f30034000 tx=0x7f9f3002e000 comp rx=0 tx=0).stop 2024-04-25T15:20:52.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 msgr2=0x7f9f3c1e85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 0x7f9f3c1e85e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00c4e0 tx=0x7f9f2c00c9a0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 shutdown_connections 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f280744e0 0x7f9f280769f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f3c158060 0x7f9f3c0cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f3c0d3300 0x7f9f3c1e95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 --2- 172.21.15.146:0/1584529894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f3c0b2cd0 0x7f9f3c1e85e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 >> 172.21.15.146:0/1584529894 conn(0x7f9f3c0b52e0 msgr2=0x7f9f3c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 shutdown_connections 2024-04-25T15:20:52.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.841+0000 7f9f42f55640 1 -- 172.21.15.146:0/1584529894 wait complete. 2024-04-25T15:20:52.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:20:52.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.962+0000 7f6e5c5ab640 1 Processor -- start 2024-04-25T15:20:52.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.962+0000 7f6e5c5ab640 1 -- start start 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.963+0000 7f6e5c5ab640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 0x7f6e540d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.963+0000 7f6e5c5ab640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e54160680 con 0x7f6e5406ee60 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.963+0000 7f6e5c5ab640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e540d20e0 con 0x7f6e5408d370 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.963+0000 7f6e5c5ab640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6e540d23c0 con 0x7f6e540b2df0 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.963+0000 7f6e59b46640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6e5406ee60 0x7f6e5408cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45890/0 (socket says 172.21.15.146:45890) 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.963+0000 7f6e59b46640 1 -- 172.21.15.146:0/477155654 learned_addr learned my addr 172.21.15.146:0/477155654 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:52.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503003646 0 0) 0x7f6e54160680 con 0x7f6e5406ee60 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5a347640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 0x7f6e540d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e40003520 con 0x7f6e5406ee60 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1368171979 0 0) 0x7f6e40003520 con 0x7f6e5406ee60 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 msgr2=0x7f6e540d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 0x7f6e540d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 >> v1:172.21.15.177:6789/0 conn(0x7f6e5408d370 legacy=0x7f6e540cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e540d26a0 con 0x7f6e5406ee60 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f6e44001670 con 0x7f6e5406ee60 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 >> v1:172.21.15.146:6789/0 conn(0x7f6e5406ee60 legacy=0x7f6e5408cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 0x7f6e400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40003f60 0x7f6e40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e40004d40 0x7f6e40005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e540d23c0 con 0x7f6e40004d40 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4000bc00 con 0x7f6e40003f60 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4000be00 con 0x7f6e540b2df0 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5ab48640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e40004d40 0x7f6e40005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e59b46640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40003f60 0x7f6e40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5a347640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 0x7f6e400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5ab48640 1 -- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 msgr2=0x7f6e400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5ab48640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b2df0 0x7f6e400039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5ab48640 1 -- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40003f60 msgr2=0x7f6e40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:52.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5ab48640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40003f60 0x7f6e40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.964+0000 7f6e5ab48640 1 -- 172.21.15.146:0/477155654 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e40003520 con 0x7f6e40004d40 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.965+0000 7f6e5ab48640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e40004d40 0x7f6e40005150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c001720 tx=0x7f6e40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f852adae16d2859 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.965+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c001370 con 0x7f6e40004d40 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.965+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e4c0052d0 con 0x7f6e40004d40 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.965+0000 7f6e59345640 1 -- 172.21.15.146:0/477155654 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c02abc0 con 0x7f6e40004d40 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/477155654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e40004d40 msgr2=0x7f6e40005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/477155654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e40004d40 0x7f6e40005150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c001720 tx=0x7f6e40005830 comp rx=0 tx=0).stop 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5ab48640 1 -- 172.21.15.146:0/477155654 reap_dead start 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/477155654 shutdown_connections 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/477155654 >> 172.21.15.146:0/477155654 conn(0x7f6e540b5360 msgr2=0x7f6e540b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:52.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/477155654 shutdown_connections 2024-04-25T15:20:52.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.966+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/477155654 wait complete. 2024-04-25T15:20:52.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.968+0000 7f6e5c5ab640 1 Processor -- start 2024-04-25T15:20:52.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.968+0000 7f6e5c5ab640 1 -- start start 2024-04-25T15:20:52.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5c5ab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40004d40 0x7f6e540b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5c5ab640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 0x7f6e54158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5c5ab640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b7710 0x7f6e540b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5c5ab640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e540b91e0 con 0x7f6e541591c0 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5c5ab640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e540d2c60 con 0x7f6e40004d40 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5c5ab640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6e540d1c70 con 0x7f6e540b7710 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 0x7f6e54158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 0x7f6e54158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55526/0 (socket says 172.21.15.146:55526) 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 -- 172.21.15.146:0/2505342800 learned_addr learned my addr 172.21.15.146:0/2505342800 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5ab48640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b7710 0x7f6e540b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e5a347640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40004d40 0x7f6e540b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 -- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b7710 msgr2=0x7f6e540b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b7710 0x7f6e540b7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 -- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40004d40 msgr2=0x7f6e540b88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.969+0000 7f6e59b46640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40004d40 0x7f6e540b88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e59b46640 1 -- 172.21.15.146:0/2505342800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e540b8060 con 0x7f6e541591c0 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e59b46640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 0x7f6e54158aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6e440072c0 tx=0x7f6e44007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400f960 con 0x7f6e541591c0 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e44018070 con 0x7f6e541591c0 2024-04-25T15:20:52.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e54163300 con 0x7f6e541591c0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400fd60 con 0x7f6e541591c0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.970+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e540c74d0 con 0x7f6e541591c0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.971+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5406a0e0 con 0x7f6e541591c0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.972+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6e4401d460 con 0x7f6e541591c0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.972+0000 7f6e4affd640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6e380744b0 0x7f6e380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.972+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f6e440956c0 con 0x7f6e541591c0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.972+0000 7f6e5a347640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6e380744b0 0x7f6e380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:52.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.973+0000 7f6e5a347640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6e380744b0 0x7f6e380769c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6e5408cc00 tx=0x7f6e5000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:52.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:52.977+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e540b8060 con 0x7f6e541591c0 2024-04-25T15:20:53.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.162+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6e540b8fc0 con 0x7f6e541591c0 2024-04-25T15:20:53.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.162+0000 7f6e4affd640 1 -- 172.21.15.146:0/2505342800 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f6e4405faf0 con 0x7f6e541591c0 2024-04-25T15:20:53.163 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6e380744b0 msgr2=0x7f6e380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6e380744b0 0x7f6e380769c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6e5408cc00 tx=0x7f6e5000b040 comp rx=0 tx=0).stop 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 msgr2=0x7f6e54158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 0x7f6e54158aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6e440072c0 tx=0x7f6e44007780 comp rx=0 tx=0).stop 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 shutdown_connections 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6e380744b0 0x7f6e380769c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6e540b7710 0x7f6e540b7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e541591c0 0x7f6e54158aa0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 --2- 172.21.15.146:0/2505342800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e40004d40 0x7f6e540b88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 >> 172.21.15.146:0/2505342800 conn(0x7f6e540b5360 msgr2=0x7f6e54163d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:53.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 shutdown_connections 2024-04-25T15:20:53.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.164+0000 7f6e5c5ab640 1 -- 172.21.15.146:0/2505342800 wait complete. 2024-04-25T15:20:53.176 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:20:53.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:53.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-25T15:20:53.206 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:20:53.206 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:20:53.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:20:53.206 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T15:20:53.264 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:20:53.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:53.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.372+0000 7f76625c0640 1 Processor -- start 2024-04-25T15:20:53.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.372+0000 7f76625c0640 1 -- start start 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f76625c0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 0x7f765c0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f76625c0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765c0b4a60 con 0x7f765c0cce10 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f76625c0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765c0b4d40 con 0x7f765c0cfbe0 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f76625c0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f765c0b5020 con 0x7f765c0d19d0 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f7660b5d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f765c0cce10 0x7f765c0cd220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45904/0 (socket says 172.21.15.146:45904) 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f7660b5d640 1 -- 172.21.15.146:0/2930557810 learned_addr learned my addr 172.21.15.146:0/2930557810 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:53.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1350077597 0 0) 0x7f765c0b4a60 con 0x7f765c0cce10 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7644003520 con 0x7f765c0cce10 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765b7fe640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 0x7f765c0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3352824368 0 0) 0x7f7644003520 con 0x7f765c0cce10 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 msgr2=0x7f765c0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 0x7f765c0d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 >> v1:172.21.15.177:6789/0 conn(0x7f765c0cfbe0 legacy=0x7f765c0cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765c0cead0 con 0x7f765c0cce10 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f76540019f0 con 0x7f765c0cce10 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 >> v1:172.21.15.146:6789/0 conn(0x7f765c0cce10 legacy=0x7f765c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76440039e0 0x7f7644003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 0x7f7644004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.374+0000 7f765affd640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7644004d40 0x7f7644005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f765c0b5020 con 0x7f7644004d40 2024-04-25T15:20:53.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f764400baf0 con 0x7f76440039e0 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f764400bcf0 con 0x7f765c0d19d0 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f7660b5d640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7644004d40 0x7f7644005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f765bfff640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76440039e0 0x7f7644003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f765b7fe640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 0x7f7644004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f7660b5d640 1 -- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 msgr2=0x7f7644004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f7660b5d640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c0d19d0 0x7f7644004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f7660b5d640 1 -- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76440039e0 msgr2=0x7f7644003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f7660b5d640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76440039e0 0x7f7644003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.375+0000 7f7660b5d640 1 -- 172.21.15.146:0/2930557810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7644003520 con 0x7f7644004d40 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f7660b5d640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7644004d40 0x7f7644005150 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f765400a860 tx=0x7f7644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4870f8d83fffc757 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:53.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765400a480 con 0x7f7644004d40 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765402f070 con 0x7f7644004d40 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f765affd640 1 -- 172.21.15.146:0/2930557810 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765402bd00 con 0x7f7644004d40 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f76625c0640 1 -- 172.21.15.146:0/2930557810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7644004d40 msgr2=0x7f7644005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f76625c0640 1 --2- 172.21.15.146:0/2930557810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7644004d40 0x7f7644005150 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f765400a860 tx=0x7f7644005830 comp rx=0 tx=0).stop 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f7660b5d640 1 -- 172.21.15.146:0/2930557810 reap_dead start 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f76625c0640 1 -- 172.21.15.146:0/2930557810 shutdown_connections 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.376+0000 7f76625c0640 1 -- 172.21.15.146:0/2930557810 >> 172.21.15.146:0/2930557810 conn(0x7f765c0b73b0 msgr2=0x7f765c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.377+0000 7f76625c0640 1 -- 172.21.15.146:0/2930557810 shutdown_connections 2024-04-25T15:20:53.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.377+0000 7f76625c0640 1 -- 172.21.15.146:0/2930557810 wait complete. 2024-04-25T15:20:53.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.378+0000 7f76625c0640 1 Processor -- start 2024-04-25T15:20:53.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.378+0000 7f76625c0640 1 -- start start 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 0x7f765c159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f765c15d170 0x7f765c1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c1f6ab0 0x7f765c0c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f765c0d1f60 con 0x7f765c15b920 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f765c0d1de0 con 0x7f765c15d170 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f765c0d20e0 con 0x7f765c1f6ab0 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 0x7f765c159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 0x7f765c159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55542/0 (socket says 172.21.15.146:55542) 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 -- 172.21.15.146:0/3608132764 learned_addr learned my addr 172.21.15.146:0/3608132764 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765b7fe640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f765c15d170 0x7f765c1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f7660b5d640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c1f6ab0 0x7f765c0c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 -- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c1f6ab0 msgr2=0x7f765c0c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c1f6ab0 0x7f765c0c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 -- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f765c15d170 msgr2=0x7f765c1f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f765c15d170 0x7f765c1f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 -- 172.21.15.146:0/3608132764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765c15a880 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765b7fe640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f765c15d170 0x7f765c1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f7660b5d640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c1f6ab0 0x7f765c0c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f765bfff640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 0x7f765c159d60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f764c00f820 tx=0x7f764c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c0050d0 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f764c018070 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765c15ab60 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.380+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c0054b0 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.381+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765c15b120 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.381+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765c15b550 con 0x7f765c15b920 2024-04-25T15:20:53.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.382+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f764c00d730 con 0x7f765c15b920 2024-04-25T15:20:53.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.382+0000 7f7658ff9640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7648074650 0x7f7648076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.383+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f764c0954d0 con 0x7f765c15b920 2024-04-25T15:20:53.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.383+0000 7f765b7fe640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7648074650 0x7f7648076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.383+0000 7f765b7fe640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7648074650 0x7f7648076b60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f765c06fae0 tx=0x7f7650008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:53.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.387+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f764c05f900 con 0x7f765c15b920 2024-04-25T15:20:53.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.600+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f765c06a0e0 con 0x7f765c15b920 2024-04-25T15:20:53.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.601+0000 7f7658ff9640 1 -- 172.21.15.146:0/3608132764 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7f764c05f200 con 0x7f765c15b920 2024-04-25T15:20:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:53.602 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:53.603 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:53.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7648074650 msgr2=0x7f7648076b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7648074650 0x7f7648076b60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f765c06fae0 tx=0x7f7650008040 comp rx=0 tx=0).stop 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 msgr2=0x7f765c159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 0x7f765c159d60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f764c00f820 tx=0x7f764c00fce0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 shutdown_connections 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7648074650 0x7f7648076b60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f765c1f6ab0 0x7f765c0c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f765c15d170 0x7f765c1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 --2- 172.21.15.146:0/3608132764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f765c15b920 0x7f765c159d60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.603+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 >> 172.21.15.146:0/3608132764 conn(0x7f765c0b73b0 msgr2=0x7f765c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.604+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 shutdown_connections 2024-04-25T15:20:53.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.604+0000 7f76625c0640 1 -- 172.21.15.146:0/3608132764 wait complete. 2024-04-25T15:20:53.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:53.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.723+0000 7fd067842640 1 Processor -- start 2024-04-25T15:20:53.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.723+0000 7fd067842640 1 -- start start 2024-04-25T15:20:53.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd067842640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0600cf040 0x7fd0600cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd067842640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd060161ab0 0x7fd060161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd067842640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0600d2460 con 0x7fd0600b8dc0 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd067842640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd0600d2a20 con 0x7fd0600cf040 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd067842640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd0600d2740 con 0x7fd060161ab0 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0600cf040 0x7fd0600cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0600cf040 0x7fd0600cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55550/0 (socket says 172.21.15.146:55550) 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 -- 172.21.15.146:0/2844284551 learned_addr learned my addr 172.21.15.146:0/2844284551 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd0655de640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd0600b8dc0 0x7fd060071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45910/0 (socket says 172.21.15.146:45910) 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd065ddf640 1 --2- 172.21.15.146:0/2844284551 >> v2:172.21.15.177:3300/0 conn(0x7fd060161ab0 0x7fd060161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 -- 172.21.15.146:0/2844284551 >> v2:172.21.15.177:3300/0 conn(0x7fd060161ab0 msgr2=0x7fd060161ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 --2- 172.21.15.146:0/2844284551 >> v2:172.21.15.177:3300/0 conn(0x7fd060161ab0 0x7fd060161ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 -- 172.21.15.146:0/2844284551 >> v1:172.21.15.146:6789/0 conn(0x7fd0600b8dc0 legacy=0x7fd060071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd064ddd640 1 -- 172.21.15.146:0/2844284551 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600ce840 con 0x7fd0600cf040 2024-04-25T15:20:53.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd057fff640 1 -- 172.21.15.146:0/2844284551 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241649245 0 0) 0x7fd0600d2460 con 0x7fd0600b8dc0 2024-04-25T15:20:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.725+0000 7fd065ddf640 1 --2- 172.21.15.146:0/2844284551 >> v2:172.21.15.177:3300/0 conn(0x7fd060161ab0 0x7fd060161ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.726+0000 7fd064ddd640 1 --2- 172.21.15.146:0/2844284551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0600cf040 0x7fd0600cf450 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd06015e540 tx=0x7fd05002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9605a1e641f611 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.726+0000 7fd057fff640 1 -- 172.21.15.146:0/2844284551 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050037070 con 0x7fd0600cf040 2024-04-25T15:20:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.726+0000 7fd057fff640 1 -- 172.21.15.146:0/2844284551 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd050032440 con 0x7fd0600cf040 2024-04-25T15:20:53.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.726+0000 7fd057fff640 1 -- 172.21.15.146:0/2844284551 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050032810 con 0x7fd0600cf040 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 -- 172.21.15.146:0/2844284551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0600cf040 msgr2=0x7fd0600cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 --2- 172.21.15.146:0/2844284551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0600cf040 0x7fd0600cf450 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd06015e540 tx=0x7fd05002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 -- 172.21.15.146:0/2844284551 shutdown_connections 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 --2- 172.21.15.146:0/2844284551 >> v2:172.21.15.177:3300/0 conn(0x7fd060161ab0 0x7fd060161ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 --2- 172.21.15.146:0/2844284551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0600cf040 0x7fd0600cf450 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 -- 172.21.15.146:0/2844284551 >> 172.21.15.146:0/2844284551 conn(0x7fd0600b5360 msgr2=0x7fd0600b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 -- 172.21.15.146:0/2844284551 shutdown_connections 2024-04-25T15:20:53.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.727+0000 7fd067842640 1 -- 172.21.15.146:0/2844284551 wait complete. 2024-04-25T15:20:53.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.729+0000 7fd067842640 1 Processor -- start 2024-04-25T15:20:53.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.729+0000 7fd067842640 1 -- start start 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd067842640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0600b8dc0 0x7fd06014d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd067842640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0600cf040 0x7fd060150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd067842640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 0x7fd0600b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd067842640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0600b2e80 con 0x7fd060161ab0 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd067842640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0600d20f0 con 0x7fd0600b8dc0 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd067842640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd0600b3000 con 0x7fd0600cf040 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 0x7fd0600b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 0x7fd0600b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55566/0 (socket says 172.21.15.146:55566) 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 -- 172.21.15.146:0/2756237632 learned_addr learned my addr 172.21.15.146:0/2756237632 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:53.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd064ddd640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0600cf040 0x7fd060150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd0655de640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0600b8dc0 0x7fd06014d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 -- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0600cf040 msgr2=0x7fd060150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0600cf040 0x7fd060150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 -- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0600b8dc0 msgr2=0x7fd06014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0600b8dc0 0x7fd06014d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 -- 172.21.15.146:0/2756237632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600cc000 con 0x7fd060161ab0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd0655de640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0600b8dc0 0x7fd06014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.730+0000 7fd065ddf640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 0x7fd0600b1a40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd05800cd20 tx=0x7fd05800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.731+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05800f680 con 0x7fd060161ab0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.731+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05800f840 con 0x7fd060161ab0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.731+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06014dbf0 con 0x7fd060161ab0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.731+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05800fc30 con 0x7fd060161ab0 2024-04-25T15:20:53.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.731+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0600ca0d0 con 0x7fd060161ab0 2024-04-25T15:20:53.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.732+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0600c9e10 con 0x7fd060161ab0 2024-04-25T15:20:53.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.733+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd0580213f0 con 0x7fd060161ab0 2024-04-25T15:20:53.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.733+0000 7fd055ffb640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0440744e0 0x7fd0440769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:53.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.733+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fd058096320 con 0x7fd060161ab0 2024-04-25T15:20:53.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.733+0000 7fd0655de640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0440744e0 0x7fd0440769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:53.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.733+0000 7fd0655de640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0440744e0 0x7fd0440769f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd05c001020 tx=0x7fd05c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:53.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.738+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd058014030 con 0x7fd060161ab0 2024-04-25T15:20:53.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.914+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd06006a0e0 con 0x7fd060161ab0 2024-04-25T15:20:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.915+0000 7fd055ffb640 1 -- 172.21.15.146:0/2756237632 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7fd0580606c0 con 0x7fd060161ab0 2024-04-25T15:20:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:53.917 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0440744e0 msgr2=0x7fd0440769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0440744e0 0x7fd0440769f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd05c001020 tx=0x7fd05c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 msgr2=0x7fd0600b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 0x7fd0600b1a40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd05800cd20 tx=0x7fd05800a650 comp rx=0 tx=0).stop 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 shutdown_connections 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0440744e0 0x7fd0440769f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060161ab0 0x7fd0600b1a40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0600cf040 0x7fd060150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 --2- 172.21.15.146:0/2756237632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0600b8dc0 0x7fd06014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:53.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 >> 172.21.15.146:0/2756237632 conn(0x7fd0600b5360 msgr2=0x7fd0600b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:53.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 shutdown_connections 2024-04-25T15:20:53.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:53.918+0000 7fd067842640 1 -- 172.21.15.146:0/2756237632 wait complete. 2024-04-25T15:20:53.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:19:44.254171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:19:44.761008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T15:19:48.481854+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T15:20:53.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:20:54.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.041+0000 7f2a9264a640 1 Processor -- start 2024-04-25T15:20:54.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.041+0000 7f2a9264a640 1 -- start start 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.042+0000 7f2a9264a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a8c0b7430 0x7f2a8c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.042+0000 7f2a9264a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 0x7f2a8c06c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a9264a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8c0b2fe0 con 0x7f2a8c0d24d0 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a9264a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8c0b3820 con 0x7f2a8c0b7430 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a9264a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c0b2850 con 0x7f2a8c06c550 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 0x7f2a8c06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8bfff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a8c0b7430 0x7f2a8c0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 0x7f2a8c06c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:43902/0 (socket says 172.21.15.146:43902) 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/3682190212 learned_addr learned my addr 172.21.15.146:0/3682190212 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/3682190212 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c0b7430 msgr2=0x7f2a8c0b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:54.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 --2- 172.21.15.146:0/3682190212 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c0b7430 0x7f2a8c0b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/3682190212 >> v1:172.21.15.177:6789/0 conn(0x7f2a8c0d24d0 legacy=0x7f2a8c0d28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.043+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/3682190212 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c0d1ff0 con 0x7f2a8c06c550 2024-04-25T15:20:54.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.044+0000 7f2a8b7fe640 1 --2- 172.21.15.146:0/3682190212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 0x7f2a8c06c960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0b31b0 tx=0x7f2a8002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6dbc612b1e677c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.044+0000 7f2a8affd640 1 -- 172.21.15.146:0/3682190212 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80037070 con 0x7f2a8c06c550 2024-04-25T15:20:54.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.044+0000 7f2a8affd640 1 -- 172.21.15.146:0/3682190212 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a80032440 con 0x7f2a8c06c550 2024-04-25T15:20:54.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.044+0000 7f2a8affd640 1 -- 172.21.15.146:0/3682190212 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80032810 con 0x7f2a8c06c550 2024-04-25T15:20:54.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 -- 172.21.15.146:0/3682190212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 msgr2=0x7f2a8c06c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 --2- 172.21.15.146:0/3682190212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 0x7f2a8c06c960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0b31b0 tx=0x7f2a8002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 -- 172.21.15.146:0/3682190212 shutdown_connections 2024-04-25T15:20:54.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 --2- 172.21.15.146:0/3682190212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06c550 0x7f2a8c06c960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 --2- 172.21.15.146:0/3682190212 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c0b7430 0x7f2a8c0b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 -- 172.21.15.146:0/3682190212 >> 172.21.15.146:0/3682190212 conn(0x7f2a8c09b2b0 msgr2=0x7f2a8c0b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 -- 172.21.15.146:0/3682190212 shutdown_connections 2024-04-25T15:20:54.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.045+0000 7f2a9264a640 1 -- 172.21.15.146:0/3682190212 wait complete. 2024-04-25T15:20:54.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.047+0000 7f2a9264a640 1 Processor -- start 2024-04-25T15:20:54.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.047+0000 7f2a9264a640 1 -- start start 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.048+0000 7f2a9264a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c06c550 0x7f2a8c159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.048+0000 7f2a9264a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 0x7f2a8c1e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.048+0000 7f2a9264a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c0d24d0 0x7f2a8c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.048+0000 7f2a9264a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c160280 con 0x7f2a8c0b7430 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.048+0000 7f2a9264a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c0b3540 con 0x7f2a8c06c550 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.048+0000 7f2a9264a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c160400 con 0x7f2a8c0d24d0 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 0x7f2a8c1e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 0x7f2a8c1e6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55578/0 (socket says 172.21.15.146:55578) 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/1171876119 learned_addr learned my addr 172.21.15.146:0/1171876119 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8bfff640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c06c550 0x7f2a8c159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a90be7640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c0d24d0 0x7f2a8c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c0d24d0 msgr2=0x7f2a8c15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c0d24d0 0x7f2a8c15a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c06c550 msgr2=0x7f2a8c159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c06c550 0x7f2a8c159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 -- 172.21.15.146:0/1171876119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c159ed0 con 0x7f2a8c0b7430 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a8b7fe640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 0x7f2a8c1e6dc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2a80000c00 tx=0x7f2a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80041030 con 0x7f2a8c0b7430 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a80030440 con 0x7f2a8c0b7430 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8c1e7940 con 0x7f2a8c0b7430 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80037040 con 0x7f2a8c0b7430 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.049+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8c1e7f00 con 0x7f2a8c0b7430 2024-04-25T15:20:54.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.050+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a8c06a0e0 con 0x7f2a8c0b7430 2024-04-25T15:20:54.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.051+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2a80044070 con 0x7f2a8c0b7430 2024-04-25T15:20:54.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.052+0000 7f2a88ff9640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a780744b0 0x7f2a780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.052+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f2a800b5c60 con 0x7f2a8c0b7430 2024-04-25T15:20:54.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.052+0000 7f2a8bfff640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a780744b0 0x7f2a780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.052+0000 7f2a8bfff640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a780744b0 0x7f2a780769c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009aa0 tx=0x7f2a7c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.056+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a8007f9b0 con 0x7f2a8c0b7430 2024-04-25T15:20:54.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.230+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a8c0b16c0 con 0x7f2a8c0b7430 2024-04-25T15:20:54.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.230+0000 7f2a88ff9640 1 -- 172.21.15.146:0/1171876119 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f2a80083cb0 con 0x7f2a8c0b7430 2024-04-25T15:20:54.231 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:20:54.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a780744b0 msgr2=0x7f2a780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a780744b0 0x7f2a780769c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009aa0 tx=0x7f2a7c009190 comp rx=0 tx=0).stop 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 msgr2=0x7f2a8c1e6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 0x7f2a8c1e6dc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2a80000c00 tx=0x7f2a800046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 shutdown_connections 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a780744b0 0x7f2a780769c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c0d24d0 0x7f2a8c15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c0b7430 0x7f2a8c1e6dc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 --2- 172.21.15.146:0/1171876119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c06c550 0x7f2a8c159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 >> 172.21.15.146:0/1171876119 conn(0x7f2a8c09b2b0 msgr2=0x7f2a8c0b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 shutdown_connections 2024-04-25T15:20:54.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.232+0000 7f2a9264a640 1 -- 172.21.15.146:0/1171876119 wait complete. 2024-04-25T15:20:54.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:20:54.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.356+0000 7fee31fb2640 1 Processor -- start 2024-04-25T15:20:54.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.356+0000 7fee31fb2640 1 -- start start 2024-04-25T15:20:54.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee31fb2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fee2c0bb450 0x7fee2c0d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee31fb2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fee2c0d0de0 0x7fee2c0b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee31fb2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c0b5230 con 0x7fee2c163c00 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee31fb2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fee2c06c8c0 con 0x7fee2c0d0de0 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee31fb2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fee2c06cba0 con 0x7fee2c0bb450 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee2bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fee2c0d0de0 0x7fee2c0b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee2bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fee2c0d0de0 0x7fee2c0b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55586/0 (socket says 172.21.15.146:55586) 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee2bfff640 1 -- 172.21.15.146:0/783701797 learned_addr learned my addr 172.21.15.146:0/783701797 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.357+0000 7fee2affd640 1 --2- 172.21.15.146:0/783701797 >> v2:172.21.15.177:3300/0 conn(0x7fee2c0bb450 0x7fee2c0d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.358+0000 7fee2bfff640 1 -- 172.21.15.146:0/783701797 >> v2:172.21.15.177:3300/0 conn(0x7fee2c0bb450 msgr2=0x7fee2c0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.358+0000 7fee2bfff640 1 --2- 172.21.15.146:0/783701797 >> v2:172.21.15.177:3300/0 conn(0x7fee2c0bb450 0x7fee2c0d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.358+0000 7fee2bfff640 1 -- 172.21.15.146:0/783701797 >> v1:172.21.15.177:6789/0 conn(0x7fee2c163c00 legacy=0x7fee2c06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.358+0000 7fee2bfff640 1 -- 172.21.15.146:0/783701797 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c06ce80 con 0x7fee2c0d0de0 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.358+0000 7fee2bfff640 1 --2- 172.21.15.146:0/783701797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0b8fb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fee24009e50 tx=0x7fee2402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9d7fc5dbed0928 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.358+0000 7fee2a7fc640 1 -- 172.21.15.146:0/783701797 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24036070 con 0x7fee2c0d0de0 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee2a7fc640 1 -- 172.21.15.146:0/783701797 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee24031600 con 0x7fee2c0d0de0 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee2a7fc640 1 -- 172.21.15.146:0/783701797 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee240319d0 con 0x7fee2c0d0de0 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 -- 172.21.15.146:0/783701797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 msgr2=0x7fee2c0b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 --2- 172.21.15.146:0/783701797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0b8fb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fee24009e50 tx=0x7fee2402a6d0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 -- 172.21.15.146:0/783701797 shutdown_connections 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 --2- 172.21.15.146:0/783701797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0b8fb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 --2- 172.21.15.146:0/783701797 >> v2:172.21.15.177:3300/0 conn(0x7fee2c0bb450 0x7fee2c0d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 -- 172.21.15.146:0/783701797 >> 172.21.15.146:0/783701797 conn(0x7fee2c0b7970 msgr2=0x7fee2c0b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 -- 172.21.15.146:0/783701797 shutdown_connections 2024-04-25T15:20:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.359+0000 7fee31fb2640 1 -- 172.21.15.146:0/783701797 wait complete. 2024-04-25T15:20:54.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.361+0000 7fee31fb2640 1 Processor -- start 2024-04-25T15:20:54.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.361+0000 7fee31fb2640 1 -- start start 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee2c0bb450 0x7fee2c0ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0ce660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2c163c00 0x7fee2c0ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c0bb860 con 0x7fee2c0d0de0 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c06c550 con 0x7fee2c163c00 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fee2c0bb9e0 con 0x7fee2c0bb450 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0ce660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55596/0 (socket says 172.21.15.146:55596) 2024-04-25T15:20:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 -- 172.21.15.146:0/1154861455 learned_addr learned my addr 172.21.15.146:0/1154861455 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2bfff640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2c163c00 0x7fee2c0ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2b7fe640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee2c0bb450 0x7fee2c0ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 -- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee2c0bb450 msgr2=0x7fee2c0ac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee2c0bb450 0x7fee2c0ac3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 -- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2c163c00 msgr2=0x7fee2c0ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2c163c00 0x7fee2c0ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 -- 172.21.15.146:0/1154861455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c0cd5c0 con 0x7fee2c0d0de0 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee2affd640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0ce660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fee2000e870 tx=0x7fee2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20017070 con 0x7fee2c0d0de0 2024-04-25T15:20:54.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2000c940 con 0x7fee2c0d0de0 2024-04-25T15:20:54.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c0cd8a0 con 0x7fee2c0d0de0 2024-04-25T15:20:54.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2000cd10 con 0x7fee2c0d0de0 2024-04-25T15:20:54.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.363+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c0cde60 con 0x7fee2c0d0de0 2024-04-25T15:20:54.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.364+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c06a0e0 con 0x7fee2c0d0de0 2024-04-25T15:20:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.365+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fee2001a070 con 0x7fee2c0d0de0 2024-04-25T15:20:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.366+0000 7fee13fff640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee180744e0 0x7fee180769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.366+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fee20095350 con 0x7fee2c0d0de0 2024-04-25T15:20:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.366+0000 7fee2b7fe640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee180744e0 0x7fee180769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.366+0000 7fee2b7fe640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee180744e0 0x7fee180769f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0016e0 tx=0x7fee1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.370+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee2005f130 con 0x7fee2c0d0de0 2024-04-25T15:20:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.547+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fee2c0d3ed0 con 0x7fee2c0d0de0 2024-04-25T15:20:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.548+0000 7fee13fff640 1 -- 172.21.15.146:0/1154861455 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fee20063430 con 0x7fee2c0d0de0 2024-04-25T15:20:54.549 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee180744e0 msgr2=0x7fee180769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee180744e0 0x7fee180769f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0016e0 tx=0x7fee1c009040 comp rx=0 tx=0).stop 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 msgr2=0x7fee2c0ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0ce660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fee2000e870 tx=0x7fee2000ed30 comp rx=0 tx=0).stop 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 shutdown_connections 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee180744e0 0x7fee180769f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2c163c00 0x7fee2c0ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee2c0d0de0 0x7fee2c0ce660 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 --2- 172.21.15.146:0/1154861455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee2c0bb450 0x7fee2c0ac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 >> 172.21.15.146:0/1154861455 conn(0x7fee2c0b7970 msgr2=0x7fee2c0b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 shutdown_connections 2024-04-25T15:20:54.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.550+0000 7fee31fb2640 1 -- 172.21.15.146:0/1154861455 wait complete. 2024-04-25T15:20:54.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:20:54.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.671+0000 7f201e500640 1 Processor -- start 2024-04-25T15:20:54.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.671+0000 7f201e500640 1 -- start start 2024-04-25T15:20:54.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f201e500640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20180b2cd0 0x7f20180d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f201e500640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f20180d3300 0x7f2018071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f201e500640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2018071f70 con 0x7f2018071b60 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f201e500640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f20180d2260 con 0x7f20180b2cd0 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f201e500640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f20180d2540 con 0x7f20180d3300 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20180b2cd0 0x7f20180d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20180b2cd0 0x7f20180d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55602/0 (socket says 172.21.15.146:55602) 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 -- 172.21.15.146:0/4045724984 learned_addr learned my addr 172.21.15.146:0/4045724984 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f20177fe640 1 --2- 172.21.15.146:0/4045724984 >> v2:172.21.15.177:3300/0 conn(0x7f20180d3300 0x7f2018071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 -- 172.21.15.146:0/4045724984 >> v2:172.21.15.177:3300/0 conn(0x7f20180d3300 msgr2=0x7f2018071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 --2- 172.21.15.146:0/4045724984 >> v2:172.21.15.177:3300/0 conn(0x7f20180d3300 0x7f2018071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 -- 172.21.15.146:0/4045724984 >> v1:172.21.15.177:6789/0 conn(0x7f2018071b60 legacy=0x7f201808cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:54.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.673+0000 7f2017fff640 1 -- 172.21.15.146:0/4045724984 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20180d2820 con 0x7f20180b2cd0 2024-04-25T15:20:54.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.674+0000 7f2017fff640 1 --2- 172.21.15.146:0/4045724984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20180b2cd0 0x7f20180d1c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2018075280 tx=0x7f200802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611237b5e1c7a6c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.674+0000 7f2016ffd640 1 -- 172.21.15.146:0/4045724984 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008004030 con 0x7f20180b2cd0 2024-04-25T15:20:54.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.674+0000 7f2016ffd640 1 -- 172.21.15.146:0/4045724984 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20080041d0 con 0x7f20180b2cd0 2024-04-25T15:20:54.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.674+0000 7f2016ffd640 1 -- 172.21.15.146:0/4045724984 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008003610 con 0x7f20180b2cd0 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 -- 172.21.15.146:0/4045724984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20180b2cd0 msgr2=0x7f20180d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 --2- 172.21.15.146:0/4045724984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20180b2cd0 0x7f20180d1c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2018075280 tx=0x7f200802a180 comp rx=0 tx=0).stop 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 -- 172.21.15.146:0/4045724984 shutdown_connections 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 --2- 172.21.15.146:0/4045724984 >> v2:172.21.15.177:3300/0 conn(0x7f20180d3300 0x7f2018071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 --2- 172.21.15.146:0/4045724984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20180b2cd0 0x7f20180d1c80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 -- 172.21.15.146:0/4045724984 >> 172.21.15.146:0/4045724984 conn(0x7f20180b52e0 msgr2=0x7f20180b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 -- 172.21.15.146:0/4045724984 shutdown_connections 2024-04-25T15:20:54.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.675+0000 7f201e500640 1 -- 172.21.15.146:0/4045724984 wait complete. 2024-04-25T15:20:54.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.676+0000 7f201e500640 1 Processor -- start 2024-04-25T15:20:54.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.677+0000 7f201e500640 1 -- start start 2024-04-25T15:20:54.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201e500640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 0x7f20181f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201e500640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20180b2cd0 0x7f20180ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201e500640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20180d3300 0x7f201806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201e500640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20180cfb70 con 0x7f2018071b60 2024-04-25T15:20:54.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201e500640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20180d2de0 con 0x7f20180b2cd0 2024-04-25T15:20:54.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201e500640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f201808dbf0 con 0x7f20180d3300 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 0x7f20181f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 0x7f20181f5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55610/0 (socket says 172.21.15.146:55610) 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 -- 172.21.15.146:0/2153868278 learned_addr learned my addr 172.21.15.146:0/2153868278 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f20177fe640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20180b2cd0 0x7f20180ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201ca9d640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20180d3300 0x7f201806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 -- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20180d3300 msgr2=0x7f201806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20180d3300 0x7f201806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 -- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20180b2cd0 msgr2=0x7f20180ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20180b2cd0 0x7f20180ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f2017fff640 1 -- 172.21.15.146:0/2153868278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20180cb450 con 0x7f2018071b60 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f20177fe640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20180b2cd0 0x7f20180ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.678+0000 7f201ca9d640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20180d3300 0x7f201806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.679+0000 7f2017fff640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 0x7f20181f5d80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2008000c00 tx=0x7f20080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.679+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008041020 con 0x7f2018071b60 2024-04-25T15:20:54.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.679+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2008003c50 con 0x7f2018071b60 2024-04-25T15:20:54.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.679+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20180cb730 con 0x7f2018071b60 2024-04-25T15:20:54.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.679+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008004030 con 0x7f2018071b60 2024-04-25T15:20:54.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.679+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20180cbcf0 con 0x7f2018071b60 2024-04-25T15:20:54.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.680+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201806a0e0 con 0x7f2018071b60 2024-04-25T15:20:54.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.680+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2008044020 con 0x7f2018071b60 2024-04-25T15:20:54.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.681+0000 7f2014ff9640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20040744e0 0x7f20040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.681+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f20080b6210 con 0x7f2018071b60 2024-04-25T15:20:54.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.681+0000 7f20177fe640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20040744e0 0x7f20040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.681+0000 7f20177fe640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20040744e0 0x7f20040769f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2018071b30 tx=0x7f200c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.686+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f200804b030 con 0x7f2018071b60 2024-04-25T15:20:54.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.861+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f201806e450 con 0x7f2018071b60 2024-04-25T15:20:54.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.861+0000 7f2014ff9640 1 -- 172.21.15.146:0/2153868278 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f2018071f70 con 0x7f2018071b60 2024-04-25T15:20:54.862 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20040744e0 msgr2=0x7f20040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20040744e0 0x7f20040769f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2018071b30 tx=0x7f200c008040 comp rx=0 tx=0).stop 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 msgr2=0x7f20181f5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 0x7f20181f5d80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2008000c00 tx=0x7f20080046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 shutdown_connections 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20040744e0 0x7f20040769f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20180d3300 0x7f201806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20180b2cd0 0x7f20180ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 --2- 172.21.15.146:0/2153868278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2018071b60 0x7f20181f5d80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 >> 172.21.15.146:0/2153868278 conn(0x7f20180b52e0 msgr2=0x7f20180b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 shutdown_connections 2024-04-25T15:20:54.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.863+0000 7f201e500640 1 -- 172.21.15.146:0/2153868278 wait complete. 2024-04-25T15:20:54.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T15:20:54.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.985+0000 7f5f454ee640 1 Processor -- start 2024-04-25T15:20:54.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.985+0000 7f5f454ee640 1 -- start start 2024-04-25T15:20:54.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f454ee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f40163c00 0x7f5f4006d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f454ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f400b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f454ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4006dea0 con 0x7f5f400bb450 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f454ee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5f400b5230 con 0x7f5f40163c00 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f454ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5f4006c8c0 con 0x7f5f400d0fb0 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3effd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f40163c00 0x7f5f4006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3e7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5f400bb450 0x7f5f400d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45916/0 (socket says 172.21.15.146:45916) 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3e7fc640 1 -- 172.21.15.146:0/2562132837 learned_addr learned my addr 172.21.15.146:0/2562132837 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:54.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3f7fe640 1 --2- 172.21.15.146:0/2562132837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f400b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3effd640 1 -- 172.21.15.146:0/2562132837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 msgr2=0x7f5f400b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3effd640 1 --2- 172.21.15.146:0/2562132837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f400b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3effd640 1 -- 172.21.15.146:0/2562132837 >> v1:172.21.15.146:6789/0 conn(0x7f5f400bb450 legacy=0x7f5f400d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.987+0000 7f5f3effd640 1 -- 172.21.15.146:0/2562132837 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4006cba0 con 0x7f5f40163c00 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.988+0000 7f5f3effd640 1 --2- 172.21.15.146:0/2562132837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f40163c00 0x7f5f4006d960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009d90 tx=0x7f5f3002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e93b5558c7ee4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.988+0000 7f5f3dffb640 1 -- 172.21.15.146:0/2562132837 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30031600 con 0x7f5f40163c00 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.988+0000 7f5f3dffb640 1 -- 172.21.15.146:0/2562132837 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f300317a0 con 0x7f5f40163c00 2024-04-25T15:20:54.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.988+0000 7f5f3dffb640 1 -- 172.21.15.146:0/2562132837 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30031b70 con 0x7f5f40163c00 2024-04-25T15:20:54.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 -- 172.21.15.146:0/2562132837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f40163c00 msgr2=0x7f5f4006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 --2- 172.21.15.146:0/2562132837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f40163c00 0x7f5f4006d960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009d90 tx=0x7f5f3002a5d0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 -- 172.21.15.146:0/2562132837 shutdown_connections 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 --2- 172.21.15.146:0/2562132837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f400b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 --2- 172.21.15.146:0/2562132837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f40163c00 0x7f5f4006d960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 -- 172.21.15.146:0/2562132837 >> 172.21.15.146:0/2562132837 conn(0x7f5f400b7970 msgr2=0x7f5f400b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 -- 172.21.15.146:0/2562132837 shutdown_connections 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.989+0000 7f5f454ee640 1 -- 172.21.15.146:0/2562132837 wait complete. 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.991+0000 7f5f454ee640 1 Processor -- start 2024-04-25T15:20:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.991+0000 7f5f454ee640 1 -- start start 2024-04-25T15:20:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f454ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 0x7f5f401fac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f454ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f40148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f454ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f40163c00 0x7f5f401f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f454ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4006c550 con 0x7f5f400bb450 2024-04-25T15:20:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f454ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4006d160 con 0x7f5f40163c00 2024-04-25T15:20:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f454ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5f400d13c0 con 0x7f5f400d0fb0 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 0x7f5f401fac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 0x7f5f401fac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55634/0 (socket says 172.21.15.146:55634) 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 -- 172.21.15.146:0/3416762569 learned_addr learned my addr 172.21.15.146:0/3416762569 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3f7fe640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f40163c00 0x7f5f401f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3e7fc640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f40148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 -- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 msgr2=0x7f5f40148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f40148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 -- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f40163c00 msgr2=0x7f5f401f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f40163c00 0x7f5f401f72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 -- 172.21.15.146:0/3416762569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f40148b80 con 0x7f5f400bb450 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3f7fe640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f40163c00 0x7f5f401f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.992+0000 7f5f3effd640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 0x7f5f401fac50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009d90 tx=0x7f5f30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.993+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3003f070 con 0x7f5f400bb450 2024-04-25T15:20:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.993+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f30030be0 con 0x7f5f400bb450 2024-04-25T15:20:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.993+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f401f6b00 con 0x7f5f400bb450 2024-04-25T15:20:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.993+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30009450 con 0x7f5f400bb450 2024-04-25T15:20:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.993+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4014b150 con 0x7f5f400bb450 2024-04-25T15:20:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.994+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f4006a0e0 con 0x7f5f400bb450 2024-04-25T15:20:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.994+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5f30044460 con 0x7f5f400bb450 2024-04-25T15:20:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.995+0000 7f5f277fe640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5f2c0744b0 0x7f5f2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.995+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5f300b71b0 con 0x7f5f400bb450 2024-04-25T15:20:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.995+0000 7f5f3e7fc640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5f2c0744b0 0x7f5f2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.995+0000 7f5f3e7fc640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5f2c0744b0 0x7f5f2c0769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5f4006ff90 tx=0x7f5f34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:54.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:54.999+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f30052020 con 0x7f5f400bb450 2024-04-25T15:20:55.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.178+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5f400b9760 con 0x7f5f400bb450 2024-04-25T15:20:55.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.179+0000 7f5f277fe640 1 -- 172.21.15.146:0/3416762569 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f5f30081070 con 0x7f5f400bb450 2024-04-25T15:20:55.180 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5f2c0744b0 msgr2=0x7f5f2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5f2c0744b0 0x7f5f2c0769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5f4006ff90 tx=0x7f5f34009040 comp rx=0 tx=0).stop 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 msgr2=0x7f5f401fac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 0x7f5f401fac50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009d90 tx=0x7f5f30004a40 comp rx=0 tx=0).stop 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 shutdown_connections 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5f2c0744b0 0x7f5f2c0769c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f40163c00 0x7f5f401f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f400d0fb0 0x7f5f40148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 --2- 172.21.15.146:0/3416762569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f400bb450 0x7f5f401fac50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 >> 172.21.15.146:0/3416762569 conn(0x7f5f400b7970 msgr2=0x7f5f400d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:55.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 shutdown_connections 2024-04-25T15:20:55.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.181+0000 7f5f454ee640 1 -- 172.21.15.146:0/3416762569 wait complete. 2024-04-25T15:20:55.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.299+0000 7f263514d640 1 Processor -- start 2024-04-25T15:20:55.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.299+0000 7f263514d640 1 -- start start 2024-04-25T15:20:55.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f263514d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f26300cde20 0x7f263016a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f263514d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26300b4a60 con 0x7f26300cfbe0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f263514d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26300b4d40 con 0x7f26300d19d0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f263514d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f26300b5020 con 0x7f26300cde20 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262ed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f26300cfbe0 0x7f26300cf260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45918/0 (socket says 172.21.15.146:45918) 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262ed9d640 1 -- 172.21.15.146:0/25437237 learned_addr learned my addr 172.21.15.146:0/25437237 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262f59e640 1 --2- 172.21.15.146:0/25437237 >> v2:172.21.15.177:3300/0 conn(0x7f26300cde20 0x7f263016a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1138164960 0 0) 0x7f26300b4a60 con 0x7f26300cfbe0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2618003520 con 0x7f26300cfbe0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3358826541 0 0) 0x7f26300b4d40 con 0x7f26300d19d0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26300b4a60 con 0x7f26300d19d0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 875486929 0 0) 0x7f2618003520 con 0x7f26300cfbe0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 >> v2:172.21.15.177:3300/0 conn(0x7f26300cde20 msgr2=0x7f263016a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 --2- 172.21.15.146:0/25437237 >> v2:172.21.15.177:3300/0 conn(0x7f26300cde20 0x7f263016a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 >> v1:172.21.15.177:6789/0 conn(0x7f26300d19d0 legacy=0x7f26300d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26300cead0 con 0x7f26300cfbe0 2024-04-25T15:20:55.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f2620001b70 con 0x7f26300cfbe0 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 >> v1:172.21.15.146:6789/0 conn(0x7f26300cfbe0 legacy=0x7f26300cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2618003b40 0x7f2618003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2618004800 0x7f2618004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26180051a0 0x7f261800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f26300b5020 con 0x7f2618004800 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f261800c140 con 0x7f2618003b40 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f261800c340 con 0x7f26180051a0 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.301+0000 7f262f59e640 1 --2- 172.21.15.146:0/25437237 >> v2:172.21.15.177:3300/0 conn(0x7f26300cde20 0x7f263016a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262e59c640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2618004800 0x7f2618004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262ed9d640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2618003b40 0x7f2618003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262f59e640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26180051a0 0x7f261800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262e59c640 1 -- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26180051a0 msgr2=0x7f261800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262e59c640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26180051a0 0x7f261800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262e59c640 1 -- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2618003b40 msgr2=0x7f2618003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262e59c640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2618003b40 0x7f2618003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262f59e640 1 -- 172.21.15.146:0/25437237 reap_dead start 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.302+0000 7f262e59c640 1 -- 172.21.15.146:0/25437237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2618003520 con 0x7f2618004800 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f262e59c640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2618004800 0x7f2618004c10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2618004780 tx=0x7f2618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3415967a1d0d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240014e0 con 0x7f2618004800 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26240052f0 con 0x7f2618004800 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f262dd9b640 1 -- 172.21.15.146:0/25437237 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262402abb0 con 0x7f2618004800 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 -- 172.21.15.146:0/25437237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2618004800 msgr2=0x7f2618004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2618004800 0x7f2618004c10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2618004780 tx=0x7f2618005830 comp rx=0 tx=0).stop 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 -- 172.21.15.146:0/25437237 shutdown_connections 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 --2- 172.21.15.146:0/25437237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2618004800 0x7f2618004c10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 -- 172.21.15.146:0/25437237 >> 172.21.15.146:0/25437237 conn(0x7f26300b73b0 msgr2=0x7f26300b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:55.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 -- 172.21.15.146:0/25437237 shutdown_connections 2024-04-25T15:20:55.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.303+0000 7f263514d640 1 -- 172.21.15.146:0/25437237 wait complete. 2024-04-25T15:20:55.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.305+0000 7f263514d640 1 Processor -- start 2024-04-25T15:20:55.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.306+0000 7f263514d640 1 -- start start 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f263514d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2618004800 0x7f26300c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f263514d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 0x7f26300b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f263514d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26300b0550 0x7f26300c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f263514d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f26300d1f60 con 0x7f2630155b70 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f263514d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f26300d1de0 con 0x7f26300b0550 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f263514d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f26300d20e0 con 0x7f2618004800 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 0x7f26300b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 0x7f26300b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55652/0 (socket says 172.21.15.146:55652) 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 -- 172.21.15.146:0/2368853140 learned_addr learned my addr 172.21.15.146:0/2368853140 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262ed9d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2618004800 0x7f26300c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262f59e640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26300b0550 0x7f26300c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 -- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2618004800 msgr2=0x7f26300c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2618004800 0x7f26300c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 -- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26300b0550 msgr2=0x7f26300c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26300b0550 0x7f26300c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 -- 172.21.15.146:0/2368853140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26300c6860 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262ed9d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2618004800 0x7f26300c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.307+0000 7f262e59c640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 0x7f26300b0010 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2624000c00 tx=0x7f26240012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.308+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2624007a50 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.308+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26240054a0 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.308+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26300c6b40 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.308+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262402be20 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.308+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26300c7100 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.309+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263006a0e0 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.309+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2624043020 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.310+0000 7f26177fe640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f261c0744e0 0x7f261c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.310+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f26240b5390 con 0x7f2630155b70 2024-04-25T15:20:55.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.310+0000 7f262ed9d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f261c0744e0 0x7f261c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.310+0000 7f262ed9d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f261c0744e0 0x7f261c0769f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2620001270 tx=0x7f2620009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.314+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f262407f730 con 0x7f2630155b70 2024-04-25T15:20:55.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.526+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26300b79d0 con 0x7f2630155b70 2024-04-25T15:20:55.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.527+0000 7f26177fe640 1 -- 172.21.15.146:0/2368853140 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7f262407f030 con 0x7f2630155b70 2024-04-25T15:20:55.528 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:55.528 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:55.528 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:55.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f261c0744e0 msgr2=0x7f261c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f261c0744e0 0x7f261c0769f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2620001270 tx=0x7f2620009040 comp rx=0 tx=0).stop 2024-04-25T15:20:55.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 msgr2=0x7f26300b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 0x7f26300b0010 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2624000c00 tx=0x7f26240012d0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 shutdown_connections 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f261c0744e0 0x7f261c0769f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26300b0550 0x7f26300c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2630155b70 0x7f26300b0010 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 --2- 172.21.15.146:0/2368853140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2618004800 0x7f26300c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 >> 172.21.15.146:0/2368853140 conn(0x7f26300b73b0 msgr2=0x7f263015fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 shutdown_connections 2024-04-25T15:20:55.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.529+0000 7f263514d640 1 -- 172.21.15.146:0/2368853140 wait complete. 2024-04-25T15:20:55.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:55.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.648+0000 7f037ccd8640 1 Processor -- start 2024-04-25T15:20:55.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.649+0000 7f037ccd8640 1 -- start start 2024-04-25T15:20:55.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037ccd8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f037806f4d0 0x7f037806fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037ccd8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f03780d1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037ccd8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03780d1cb0 0x7f03780b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037ccd8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f03780d0e00 con 0x7f03780d1cb0 2024-04-25T15:20:55.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037ccd8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f03780b2b50 con 0x7f037806f4d0 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037ccd8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f03780b2e30 con 0x7f0378160020 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03780d1cb0 0x7f03780b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03780d1cb0 0x7f03780b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55656/0 (socket says 172.21.15.146:55656) 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 -- 172.21.15.146:0/580278604 learned_addr learned my addr 172.21.15.146:0/580278604 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0375d9b640 1 --2- 172.21.15.146:0/580278604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f03780d1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f037659c640 1 --2- 172.21.15.146:0/580278604 >> v2:172.21.15.177:3300/0 conn(0x7f037806f4d0 0x7f037806fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 -- 172.21.15.146:0/580278604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 msgr2=0x7f03780d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 --2- 172.21.15.146:0/580278604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f03780d1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 -- 172.21.15.146:0/580278604 >> v2:172.21.15.177:3300/0 conn(0x7f037806f4d0 msgr2=0x7f037806fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.650+0000 7f0376d9d640 1 --2- 172.21.15.146:0/580278604 >> v2:172.21.15.177:3300/0 conn(0x7f037806f4d0 0x7f037806fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.651+0000 7f0376d9d640 1 -- 172.21.15.146:0/580278604 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037806f260 con 0x7f03780d1cb0 2024-04-25T15:20:55.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.651+0000 7f0376d9d640 1 --2- 172.21.15.146:0/580278604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03780d1cb0 0x7f03780b5240 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f037806dc60 tx=0x7f037002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ed580938ac6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.651+0000 7f037559a640 1 -- 172.21.15.146:0/580278604 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370037070 con 0x7f03780d1cb0 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.651+0000 7f037559a640 1 -- 172.21.15.146:0/580278604 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0370032440 con 0x7f03780d1cb0 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.651+0000 7f037559a640 1 -- 172.21.15.146:0/580278604 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370032810 con 0x7f03780d1cb0 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 -- 172.21.15.146:0/580278604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03780d1cb0 msgr2=0x7f03780b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 --2- 172.21.15.146:0/580278604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03780d1cb0 0x7f03780b5240 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f037806dc60 tx=0x7f037002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 -- 172.21.15.146:0/580278604 shutdown_connections 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 --2- 172.21.15.146:0/580278604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03780d1cb0 0x7f03780b5240 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 --2- 172.21.15.146:0/580278604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f03780d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 --2- 172.21.15.146:0/580278604 >> v2:172.21.15.177:3300/0 conn(0x7f037806f4d0 0x7f037806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 -- 172.21.15.146:0/580278604 >> 172.21.15.146:0/580278604 conn(0x7f03780d0240 msgr2=0x7f03780d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 -- 172.21.15.146:0/580278604 shutdown_connections 2024-04-25T15:20:55.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.652+0000 7f037ccd8640 1 -- 172.21.15.146:0/580278604 wait complete. 2024-04-25T15:20:55.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.654+0000 7f037ccd8640 1 Processor -- start 2024-04-25T15:20:55.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.654+0000 7f037ccd8640 1 -- start start 2024-04-25T15:20:55.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037ccd8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 0x7f03781630d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037ccd8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03780d1cb0 0x7f037815c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037ccd8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f037815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037ccd8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03781605b0 con 0x7f037806f4d0 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037ccd8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0378160430 con 0x7f03780d1cb0 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037ccd8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f03780b5780 con 0x7f0378160020 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.655+0000 7f037659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 0x7f03781630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 0x7f03781630d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55664/0 (socket says 172.21.15.146:55664) 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 -- 172.21.15.146:0/4256851342 learned_addr learned my addr 172.21.15.146:0/4256851342 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f0375d9b640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03780d1cb0 0x7f037815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f0376d9d640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f037815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 -- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 msgr2=0x7f037815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f037815c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 -- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03780d1cb0 msgr2=0x7f037815c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03780d1cb0 0x7f037815c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 -- 172.21.15.146:0/4256851342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037815d650 con 0x7f037806f4d0 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037659c640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 0x7f03781630d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0368009d10 tx=0x7f036800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368018020 con 0x7f037806f4d0 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0368014480 con 0x7f037806f4d0 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03781618e0 con 0x7f037806f4d0 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368014830 con 0x7f037806f4d0 2024-04-25T15:20:55.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.656+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0378161ea0 con 0x7f037806f4d0 2024-04-25T15:20:55.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.657+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037806c9e0 con 0x7f037806f4d0 2024-04-25T15:20:55.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.658+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0368005d10 con 0x7f037806f4d0 2024-04-25T15:20:55.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.658+0000 7f0366ffd640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03580744b0 0x7f03580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.659+0000 7f0375d9b640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03580744b0 0x7f03580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.659+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f0368096300 con 0x7f037806f4d0 2024-04-25T15:20:55.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.659+0000 7f0375d9b640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03580744b0 0x7f03580769c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f03780ca2f0 tx=0x7f036c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.663+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f037806c9e0 con 0x7f037806f4d0 2024-04-25T15:20:55.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.843+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03780b5ef0 con 0x7f037806f4d0 2024-04-25T15:20:55.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.844+0000 7f0366ffd640 1 -- 172.21.15.146:0/4256851342 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f03780b5ef0 con 0x7f037806f4d0 2024-04-25T15:20:55.845 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:55.845 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:55.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03580744b0 msgr2=0x7f03580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03580744b0 0x7f03580769c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f03780ca2f0 tx=0x7f036c009190 comp rx=0 tx=0).stop 2024-04-25T15:20:55.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 msgr2=0x7f03781630d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 0x7f03781630d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0368009d10 tx=0x7f036800c490 comp rx=0 tx=0).stop 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 shutdown_connections 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f03580744b0 0x7f03580769c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0378160020 0x7f037815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03780d1cb0 0x7f037815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 --2- 172.21.15.146:0/4256851342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f037806f4d0 0x7f03781630d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 >> 172.21.15.146:0/4256851342 conn(0x7f03780d0240 msgr2=0x7f03780c99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 shutdown_connections 2024-04-25T15:20:55.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.846+0000 7f037ccd8640 1 -- 172.21.15.146:0/4256851342 wait complete. 2024-04-25T15:20:55.859 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:20:55.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:55.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.966+0000 7ff218613640 1 Processor -- start 2024-04-25T15:20:55.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.966+0000 7ff218613640 1 -- start start 2024-04-25T15:20:55.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.967+0000 7ff218613640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff2100d4f60 0x7ff2100d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.967+0000 7ff218613640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 0x7ff2100d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff218613640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2100b9200 con 0x7ff210163d80 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff218613640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff21006c550 con 0x7ff2100d4f60 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff218613640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff21006c830 con 0x7ff2100d0020 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff216bb0640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff210163d80 0x7ff210164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45934/0 (socket says 172.21.15.146:45934) 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff216bb0640 1 -- 172.21.15.146:0/1682599843 learned_addr learned my addr 172.21.15.146:0/1682599843 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff215bae640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 0x7ff2100d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2163af640 1 --2- 172.21.15.146:0/1682599843 >> v2:172.21.15.177:3300/0 conn(0x7ff2100d4f60 0x7ff2100d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041577143 0 0) 0x7ff208001490 con 0x7ff210163d80 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1fc003520 con 0x7ff210163d80 2024-04-25T15:20:55.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 31888523 0 0) 0x7ff1fc003520 con 0x7ff210163d80 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 msgr2=0x7ff2100d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 0x7ff2100d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 >> v2:172.21.15.177:3300/0 conn(0x7ff2100d4f60 msgr2=0x7ff2100d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 --2- 172.21.15.146:0/1682599843 >> v2:172.21.15.177:3300/0 conn(0x7ff2100d4f60 0x7ff2100d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2100d0970 con 0x7ff210163d80 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7ff208001bc0 con 0x7ff210163d80 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 >> v1:172.21.15.146:6789/0 conn(0x7ff210163d80 legacy=0x7ff210164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0039b0 0x7ff1fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 0x7ff1fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1fc004d40 0x7ff1fc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff21006c550 con 0x7ff1fc0039b0 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff21006c830 con 0x7ff1fc004d40 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff1fc00baf0 con 0x7ff2100d0020 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.968+0000 7ff2163af640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0039b0 0x7ff1fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff215bae640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 0x7ff1fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff216bb0640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1fc004d40 0x7ff1fc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff2163af640 1 -- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 msgr2=0x7ff1fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff2163af640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2100d0020 0x7ff1fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff2163af640 1 -- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1fc004d40 msgr2=0x7ff1fc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff2163af640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1fc004d40 0x7ff1fc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff2163af640 1 -- 172.21.15.146:0/1682599843 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc003520 con 0x7ff1fc0039b0 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff216bb0640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1fc004d40 0x7ff1fc005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.969+0000 7ff2163af640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0039b0 0x7ff1fc003e20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff210163ce0 tx=0x7ff1fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927b924d939854b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c036070 con 0x7ff1fc0039b0 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20c031400 con 0x7ff1fc0039b0 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff2153ad640 1 -- 172.21.15.146:0/1682599843 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c009690 con 0x7ff1fc0039b0 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff218613640 1 -- 172.21.15.146:0/1682599843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0039b0 msgr2=0x7ff1fc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff218613640 1 --2- 172.21.15.146:0/1682599843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0039b0 0x7ff1fc003e20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff210163ce0 tx=0x7ff1fc005830 comp rx=0 tx=0).stop 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff216bb0640 1 -- 172.21.15.146:0/1682599843 reap_dead start 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff218613640 1 -- 172.21.15.146:0/1682599843 shutdown_connections 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff218613640 1 -- 172.21.15.146:0/1682599843 >> 172.21.15.146:0/1682599843 conn(0x7ff2100d2450 msgr2=0x7ff2100d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff218613640 1 -- 172.21.15.146:0/1682599843 shutdown_connections 2024-04-25T15:20:55.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.970+0000 7ff218613640 1 -- 172.21.15.146:0/1682599843 wait complete. 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.972+0000 7ff218613640 1 Processor -- start 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.972+0000 7ff218613640 1 -- start start 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.973+0000 7ff218613640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1fc0039b0 0x7ff2101f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.973+0000 7ff218613640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 0x7ff2101484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.973+0000 7ff218613640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2101489f0 0x7ff210148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.973+0000 7ff218613640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100d15f0 con 0x7ff21014f0a0 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.973+0000 7ff218613640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100d1070 con 0x7ff2101489f0 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.973+0000 7ff218613640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff2100b5150 con 0x7ff1fc0039b0 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 0x7ff2101484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 0x7ff2101484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55678/0 (socket says 172.21.15.146:55678) 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 -- 172.21.15.146:0/72479927 learned_addr learned my addr 172.21.15.146:0/72479927 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:55.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff2163af640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1fc0039b0 0x7ff2101f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff216bb0640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2101489f0 0x7ff210148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 -- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1fc0039b0 msgr2=0x7ff2101f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1fc0039b0 0x7ff2101f63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 -- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2101489f0 msgr2=0x7ff210148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2101489f0 0x7ff210148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 -- 172.21.15.146:0/72479927 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2100c9710 con 0x7ff21014f0a0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff216bb0640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2101489f0 0x7ff210148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff215bae640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 0x7ff2101484b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff2000119d0 tx=0x7ff200011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20000fa90 con 0x7ff21014f0a0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20000fc30 con 0x7ff21014f0a0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2100c99f0 con 0x7ff21014f0a0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200014440 con 0x7ff21014f0a0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.974+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2101499d0 con 0x7ff21014f0a0 2024-04-25T15:20:55.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.975+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff21006a0e0 con 0x7ff21014f0a0 2024-04-25T15:20:55.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.976+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff200012040 con 0x7ff21014f0a0 2024-04-25T15:20:55.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.976+0000 7ff206ffd640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1f40744e0 0x7ff1f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:55.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.976+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7ff200095b10 con 0x7ff21014f0a0 2024-04-25T15:20:55.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.977+0000 7ff2163af640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1f40744e0 0x7ff1f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:55.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.977+0000 7ff2163af640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1f40744e0 0x7ff1f40769f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff20c033000 tx=0x7ff20c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:55.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:55.981+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff200019030 con 0x7ff21014f0a0 2024-04-25T15:20:56.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.191+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2100d4100 con 0x7ff21014f0a0 2024-04-25T15:20:56.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.192+0000 7ff206ffd640 1 -- 172.21.15.146:0/72479927 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7ff20005feb0 con 0x7ff21014f0a0 2024-04-25T15:20:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:56.193 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:56.193 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1f40744e0 msgr2=0x7ff1f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1f40744e0 0x7ff1f40769f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff20c033000 tx=0x7ff20c008030 comp rx=0 tx=0).stop 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 msgr2=0x7ff2101484b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 0x7ff2101484b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff2000119d0 tx=0x7ff200011e90 comp rx=0 tx=0).stop 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 shutdown_connections 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff1f40744e0 0x7ff1f40769f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2101489f0 0x7ff210148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21014f0a0 0x7ff2101484b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 --2- 172.21.15.146:0/72479927 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1fc0039b0 0x7ff2101f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 >> 172.21.15.146:0/72479927 conn(0x7ff2100d2450 msgr2=0x7ff2100b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:56.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 shutdown_connections 2024-04-25T15:20:56.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.194+0000 7ff218613640 1 -- 172.21.15.146:0/72479927 wait complete. 2024-04-25T15:20:56.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:20:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.314+0000 7fcb1270e640 1 Processor -- start 2024-04-25T15:20:56.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.314+0000 7fcb1270e640 1 -- start start 2024-04-25T15:20:56.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb1270e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcb0c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb1270e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c160410 con 0x7fcb0c0c6cc0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb1270e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c0b2d50 con 0x7fcb0c0c68b0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb1270e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c06c8c0 con 0x7fcb0c06da10 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb10cab640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcb0c0c6cc0 0x7fcb0c164bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45944/0 (socket says 172.21.15.146:45944) 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb10cab640 1 -- 172.21.15.146:0/1407288307 learned_addr learned my addr 172.21.15.146:0/1407288307 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0bfff640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcb0c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1272976872 0 0) 0x7fcb0c160410 con 0x7fcb0c0c6cc0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaf4003520 con 0x7fcb0c0c6cc0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729926195 0 0) 0x7fcb0c0b2d50 con 0x7fcb0c0c68b0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb0c160410 con 0x7fcb0c0c68b0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1215078968 0 0) 0x7fcaf4003520 con 0x7fcb0c0c6cc0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 msgr2=0x7fcb0c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcb0c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 >> v1:172.21.15.177:6789/0 conn(0x7fcb0c0c68b0 legacy=0x7fcb0c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0b3030 con 0x7fcb0c0c6cc0 2024-04-25T15:20:56.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fcb04001970 con 0x7fcb0c0c6cc0 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.316+0000 7fcb0bfff640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcb0c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 >> v1:172.21.15.146:6789/0 conn(0x7fcb0c0c6cc0 legacy=0x7fcb0c164bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4003b70 0x7fcaf4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcaf40052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c06c8c0 con 0x7fcaf4004800 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf400bba0 con 0x7fcaf4003b70 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf400bda0 con 0x7fcb0c06da10 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0b7fe640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0bfff640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4003b70 0x7fcaf4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb10cab640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcaf40052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0b7fe640 1 -- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 msgr2=0x7fcaf40052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0b7fe640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c06da10 0x7fcaf40052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0b7fe640 1 -- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4003b70 msgr2=0x7fcaf4003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0b7fe640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4003b70 0x7fcaf4003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.317+0000 7fcb0b7fe640 1 -- 172.21.15.146:0/1407288307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf4003520 con 0x7fcaf4004800 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb0b7fe640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcb000018d0 tx=0x7fcaf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adfbde542dab8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000014f0 con 0x7fcaf4004800 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb000052d0 con 0x7fcaf4004800 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb0affd640 1 -- 172.21.15.146:0/1407288307 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0002aca0 con 0x7fcaf4004800 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb1270e640 1 -- 172.21.15.146:0/1407288307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcaf4004800 msgr2=0x7fcaf4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb1270e640 1 --2- 172.21.15.146:0/1407288307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcb000018d0 tx=0x7fcaf4005830 comp rx=0 tx=0).stop 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.318+0000 7fcb10cab640 1 -- 172.21.15.146:0/1407288307 reap_dead start 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.319+0000 7fcb1270e640 1 -- 172.21.15.146:0/1407288307 shutdown_connections 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.319+0000 7fcb1270e640 1 -- 172.21.15.146:0/1407288307 >> 172.21.15.146:0/1407288307 conn(0x7fcb0c09b2b0 msgr2=0x7fcb0c0b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.319+0000 7fcb1270e640 1 -- 172.21.15.146:0/1407288307 shutdown_connections 2024-04-25T15:20:56.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.319+0000 7fcb1270e640 1 -- 172.21.15.146:0/1407288307 wait complete. 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.320+0000 7fcb1270e640 1 Processor -- start 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.321+0000 7fcb1270e640 1 -- start start 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb1270e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4004800 0x7fcb0c1f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb1270e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c15b460 0x7fcb0c15a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb1270e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 0x7fcb0c1f9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb1270e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c1652d0 con 0x7fcb0c164ba0 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb1270e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c06c550 con 0x7fcaf4004800 2024-04-25T15:20:56.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb1270e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c165450 con 0x7fcb0c15b460 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 0x7fcb0c1f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 0x7fcb0c1f9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55698/0 (socket says 172.21.15.146:55698) 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 -- 172.21.15.146:0/2787343589 learned_addr learned my addr 172.21.15.146:0/2787343589 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb0b7fe640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c15b460 0x7fcb0c15a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb0bfff640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4004800 0x7fcb0c1f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 -- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c15b460 msgr2=0x7fcb0c15a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c15b460 0x7fcb0c15a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 -- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4004800 msgr2=0x7fcb0c1f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4004800 0x7fcb0c1f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 -- 172.21.15.146:0/2787343589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c09f570 con 0x7fcb0c164ba0 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.322+0000 7fcb10cab640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 0x7fcb0c1f9540 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0735a0 tx=0x7fcb04007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.323+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400aa70 con 0x7fcb0c164ba0 2024-04-25T15:20:56.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.323+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04018070 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.323+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c1f5f50 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.323+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb040133f0 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.323+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c1f6210 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.324+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c06a0e0 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.324+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb04013590 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.325+0000 7fcb08ff9640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcaf80744e0 0x7fcaf80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.325+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fcb040959d0 con 0x7fcb0c164ba0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.325+0000 7fcb0bfff640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcaf80744e0 0x7fcaf80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.325+0000 7fcb0bfff640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcaf80744e0 0x7fcaf80769f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009560 tx=0x7fcafc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.329+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb0405fe90 con 0x7fcb0c164ba0 2024-04-25T15:20:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.539+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb0c0b7130 con 0x7fcb0c164ba0 2024-04-25T15:20:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.540+0000 7fcb08ff9640 1 -- 172.21.15.146:0/2787343589 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5189 (secure 0 0 0) 0x7fcb0405f790 con 0x7fcb0c164ba0 2024-04-25T15:20:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:56.540 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2994811659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2994811659},{"type":"v1","addr":"172.21.15.146:6835","nonce":2994811659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3123339589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3123339589},{"type":"v1","addr":"172.21.15.146:6837","nonce":3123339589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2259653455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2259653455},{"type":"v1","addr":"172.21.15.177:6835","nonce":2259653455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:19:45.824153+0000","modified":"2024-04-25T15:20:07.899992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6834/3903573056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3903573056},{"type":"v1","addr":"172.21.15.177:6834","nonce":3903573056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-25T15:20:56.541 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcaf80744e0 msgr2=0x7fcaf80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcaf80744e0 0x7fcaf80769f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009560 tx=0x7fcafc008040 comp rx=0 tx=0).stop 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 msgr2=0x7fcb0c1f9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 0x7fcb0c1f9540 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0735a0 tx=0x7fcb04007bd0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 shutdown_connections 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcaf80744e0 0x7fcaf80769f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb0c164ba0 0x7fcb0c1f9540 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb0c15b460 0x7fcb0c15a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 --2- 172.21.15.146:0/2787343589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcaf4004800 0x7fcb0c1f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 >> 172.21.15.146:0/2787343589 conn(0x7fcb0c09b2b0 msgr2=0x7fcb0c163b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 shutdown_connections 2024-04-25T15:20:56.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.542+0000 7fcb1270e640 1 -- 172.21.15.146:0/2787343589 wait complete. 2024-04-25T15:20:56.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:20:56.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.661+0000 7f571dee7640 1 Processor -- start 2024-04-25T15:20:56.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.662+0000 7f571dee7640 1 -- start start 2024-04-25T15:20:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f571dee7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f571806ee60 0x7f57180b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f571dee7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5718163eb0 0x7f57181642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f571dee7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57180b8eb0 con 0x7f5718164800 2024-04-25T15:20:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f571dee7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f571808d730 con 0x7f5718163eb0 2024-04-25T15:20:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f571dee7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f571806f440 con 0x7f571806ee60 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5718163eb0 0x7f57181642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5718163eb0 0x7f57181642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55706/0 (socket says 172.21.15.146:55706) 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5717fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5718164800 0x7f5718160830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45950/0 (socket says 172.21.15.146:45950) 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 -- 172.21.15.146:0/68376142 learned_addr learned my addr 172.21.15.146:0/68376142 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f57177fe640 1 --2- 172.21.15.146:0/68376142 >> v2:172.21.15.177:3300/0 conn(0x7f571806ee60 0x7f57180b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 -- 172.21.15.146:0/68376142 >> v2:172.21.15.177:3300/0 conn(0x7f571806ee60 msgr2=0x7f57180b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 --2- 172.21.15.146:0/68376142 >> v2:172.21.15.177:3300/0 conn(0x7f571806ee60 0x7f57180b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 -- 172.21.15.146:0/68376142 >> v1:172.21.15.146:6789/0 conn(0x7f5718164800 legacy=0x7f5718160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f5716ffd640 1 -- 172.21.15.146:0/68376142 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571808ce00 con 0x7f5718163eb0 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f57167fc640 1 -- 172.21.15.146:0/68376142 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431172072 0 0) 0x7f57180b8eb0 con 0x7f5718164800 2024-04-25T15:20:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.663+0000 7f57177fe640 1 --2- 172.21.15.146:0/68376142 >> v2:172.21.15.177:3300/0 conn(0x7f571806ee60 0x7f57180b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:20:56.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.664+0000 7f5716ffd640 1 --2- 172.21.15.146:0/68376142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718163eb0 0x7f57181642c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f571808c9d0 tx=0x7f570c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9872e0bfda399f6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.664+0000 7f57167fc640 1 -- 172.21.15.146:0/68376142 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c037070 con 0x7f5718163eb0 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.664+0000 7f57167fc640 1 -- 172.21.15.146:0/68376142 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c032440 con 0x7f5718163eb0 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.664+0000 7f57167fc640 1 -- 172.21.15.146:0/68376142 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c032810 con 0x7f5718163eb0 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 -- 172.21.15.146:0/68376142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718163eb0 msgr2=0x7f57181642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 --2- 172.21.15.146:0/68376142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718163eb0 0x7f57181642c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f571808c9d0 tx=0x7f570c02a270 comp rx=0 tx=0).stop 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 -- 172.21.15.146:0/68376142 shutdown_connections 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 --2- 172.21.15.146:0/68376142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718163eb0 0x7f57181642c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 --2- 172.21.15.146:0/68376142 >> v2:172.21.15.177:3300/0 conn(0x7f571806ee60 0x7f57180b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 -- 172.21.15.146:0/68376142 >> 172.21.15.146:0/68376142 conn(0x7f57180b52e0 msgr2=0x7f57180b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 -- 172.21.15.146:0/68376142 shutdown_connections 2024-04-25T15:20:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.665+0000 7f571dee7640 1 -- 172.21.15.146:0/68376142 wait complete. 2024-04-25T15:20:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.667+0000 7f571dee7640 1 Processor -- start 2024-04-25T15:20:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.667+0000 7f571dee7640 1 -- start start 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f571dee7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f571806ee60 0x7f5718152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f571dee7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5718163eb0 0x7f571814bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f571dee7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 0x7f57180b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f571dee7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f571808dbc0 con 0x7f5718164800 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f571dee7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f571808d3c0 con 0x7f5718163eb0 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f571dee7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f57180d2b70 con 0x7f571806ee60 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.668+0000 7f5717fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 0x7f57180b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 0x7f57180b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55710/0 (socket says 172.21.15.146:55710) 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 -- 172.21.15.146:0/2604270004 learned_addr learned my addr 172.21.15.146:0/2604270004 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5716ffd640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5718163eb0 0x7f571814bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f57177fe640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f571806ee60 0x7f5718152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 -- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f571806ee60 msgr2=0x7f5718152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f571806ee60 0x7f5718152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 -- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5718163eb0 msgr2=0x7f571814bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5718163eb0 0x7f571814bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 -- 172.21.15.146:0/2604270004 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5718152970 con 0x7f5718164800 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f5717fff640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 0x7f57180b3510 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f571000c730 tx=0x7f571000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710018070 con 0x7f5718164800 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571000ce40 con 0x7f5718164800 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571814fdc0 con 0x7f5718164800 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571000a880 con 0x7f5718164800 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.669+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5718150380 con 0x7f5718164800 2024-04-25T15:20:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.670+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f571806a0e0 con 0x7f5718164800 2024-04-25T15:20:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.671+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f571001b070 con 0x7f5718164800 2024-04-25T15:20:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.671+0000 7f56fffff640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f57040744b0 0x7f57040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.671+0000 7f57177fe640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f57040744b0 0x7f57040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.671+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7f5710094af0 con 0x7f5718164800 2024-04-25T15:20:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.672+0000 7f57177fe640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f57040744b0 0x7f57040769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57080045f0 tx=0x7f5708008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.676+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f571005e840 con 0x7f5718164800 2024-04-25T15:20:56.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.855+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57180b61f0 con 0x7f5718164800 2024-04-25T15:20:56.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.857+0000 7f56fffff640 1 -- 172.21.15.146:0/2604270004 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18920 (secure 0 0 0) 0x7f5710062b40 con 0x7f5718164800 2024-04-25T15:20:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:20:56.858 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":27,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:20:27.635663+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:19:44.254171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:19:44.761008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T15:19:48.481854+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f57040744b0 msgr2=0x7f57040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f57040744b0 0x7f57040769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57080045f0 tx=0x7f5708008040 comp rx=0 tx=0).stop 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 msgr2=0x7f57180b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 0x7f57180b3510 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f571000c730 tx=0x7f571000cbf0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 shutdown_connections 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f57040744b0 0x7f57040769c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5718164800 0x7f57180b3510 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5718163eb0 0x7f571814bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 --2- 172.21.15.146:0/2604270004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f571806ee60 0x7f5718152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 >> 172.21.15.146:0/2604270004 conn(0x7f57180b52e0 msgr2=0x7f57180684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 shutdown_connections 2024-04-25T15:20:56.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.859+0000 7f571dee7640 1 -- 172.21.15.146:0/2604270004 wait complete. 2024-04-25T15:20:56.872 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:20:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.981+0000 7fcdbebc5640 1 Processor -- start 2024-04-25T15:20:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.981+0000 7fcdbebc5640 1 -- start start 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.982+0000 7fcdbebc5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb80d2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.982+0000 7fcdbebc5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcdb80d2aa0 0x7fcdb806def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.982+0000 7fcdbebc5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb80b3fc0 con 0x7fcdb815cfd0 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.982+0000 7fcdbebc5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcdb80cefe0 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.982+0000 7fcdbebc5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb80d2eb0 con 0x7fcdb80b3bb0 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb80d2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdbd162640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcdb80d2aa0 0x7fcdb806def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb80d2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:44076/0 (socket says 172.21.15.146:44076) 2024-04-25T15:20:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 -- 172.21.15.146:0/1904438909 learned_addr learned my addr 172.21.15.146:0/1904438909 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 -- 172.21.15.146:0/1904438909 >> v2:172.21.15.177:3300/0 conn(0x7fcdb80d2aa0 msgr2=0x7fcdb806def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 --2- 172.21.15.146:0/1904438909 >> v2:172.21.15.177:3300/0 conn(0x7fcdb80d2aa0 0x7fcdb806def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 -- 172.21.15.146:0/1904438909 >> v1:172.21.15.177:6789/0 conn(0x7fcdb815cfd0 legacy=0x7fcdb809bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdb7fff640 1 -- 172.21.15.146:0/1904438909 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb80cfff0 con 0x7fcdb80b3bb0 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.983+0000 7fcdbd162640 1 --2- 172.21.15.146:0/1904438909 >> v2:172.21.15.177:3300/0 conn(0x7fcdb80d2aa0 0x7fcdb806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.984+0000 7fcdb7fff640 1 --2- 172.21.15.146:0/1904438909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb80d2560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009870 tx=0x7fcda402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1140de01ae5b4448 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.984+0000 7fcdb77fe640 1 -- 172.21.15.146:0/1904438909 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4037070 con 0x7fcdb80b3bb0 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.984+0000 7fcdb77fe640 1 -- 172.21.15.146:0/1904438909 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda4032440 con 0x7fcdb80b3bb0 2024-04-25T15:20:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.984+0000 7fcdb77fe640 1 -- 172.21.15.146:0/1904438909 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4032810 con 0x7fcdb80b3bb0 2024-04-25T15:20:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 -- 172.21.15.146:0/1904438909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 msgr2=0x7fcdb80d2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/1904438909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb80d2560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009870 tx=0x7fcda402a270 comp rx=0 tx=0).stop 2024-04-25T15:20:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 -- 172.21.15.146:0/1904438909 shutdown_connections 2024-04-25T15:20:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/1904438909 >> v2:172.21.15.177:3300/0 conn(0x7fcdb80d2aa0 0x7fcdb806def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/1904438909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb80d2560 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 -- 172.21.15.146:0/1904438909 >> 172.21.15.146:0/1904438909 conn(0x7fcdb80b7530 msgr2=0x7fcdb80b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 -- 172.21.15.146:0/1904438909 shutdown_connections 2024-04-25T15:20:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.985+0000 7fcdbebc5640 1 -- 172.21.15.146:0/1904438909 wait complete. 2024-04-25T15:20:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.986+0000 7fcdbebc5640 1 Processor -- start 2024-04-25T15:20:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.986+0000 7fcdbebc5640 1 -- start start 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbebc5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbebc5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 0x7fcdb815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbebc5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdb814e210 0x7fcdb80c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbebc5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb80cf7f0 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbebc5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb80cf510 con 0x7fcdb814e210 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbebc5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb80cfad0 con 0x7fcdb80b3bb0 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 0x7fcdb815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 0x7fcdb815f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55720/0 (socket says 172.21.15.146:55720) 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 -- 172.21.15.146:0/4036516241 learned_addr learned my addr 172.21.15.146:0/4036516241 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbd162640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdb814e210 0x7fcdb80c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdbc961640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 -- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 msgr2=0x7fcdb815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb815d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 -- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdb814e210 msgr2=0x7fcdb80c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdb814e210 0x7fcdb80c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 -- 172.21.15.146:0/4036516241 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb80ca060 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.988+0000 7fcdb7fff640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 0x7fcdb815f9a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcda4000c00 tx=0x7fcda40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.989+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4037040 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.989+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda4042070 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.989+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda4030880 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.989+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb8160310 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.989+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81605f0 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.990+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb806a0e0 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.990+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcda4030a20 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.991+0000 7fcdb57fa640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd9c0744b0 0x7fcd9c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.991+0000 7fcdbc961640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd9c0744b0 0x7fcd9c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.991+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6714+0+0 (secure 0 0 0) 0x7fcda40b5f60 con 0x7fcdb80d2aa0 2024-04-25T15:20:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.992+0000 7fcdbc961640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd9c0744b0 0x7fcd9c0769c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcdac007fd0 tx=0x7fcdac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:56.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:56.996+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcda4080300 con 0x7fcdb80d2aa0 2024-04-25T15:20:57.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.207+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcdb80c82b0 con 0x7fcdb80d2aa0 2024-04-25T15:20:57.208 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-04-25T15:20:57.207+0000 7f79638be640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:20:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.209+0000 7fcdb57fa640 1 -- 172.21.15.146:0/4036516241 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fcda407fc00 con 0x7fcdb80d2aa0 2024-04-25T15:20:57.210 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:20:57.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.212+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd9c0744b0 msgr2=0x7fcd9c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:57.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.212+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd9c0744b0 0x7fcd9c0769c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcdac007fd0 tx=0x7fcdac009040 comp rx=0 tx=0).stop 2024-04-25T15:20:57.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.212+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 msgr2=0x7fcdb815f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:57.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.212+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 0x7fcdb815f9a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fcda4000c00 tx=0x7fcda40046c0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 shutdown_connections 2024-04-25T15:20:57.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd9c0744b0 0x7fcd9c0769c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdb814e210 0x7fcdb80c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80d2aa0 0x7fcdb815f9a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 --2- 172.21.15.146:0/4036516241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcdb80b3bb0 0x7fcdb815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 >> 172.21.15.146:0/4036516241 conn(0x7fcdb80b7530 msgr2=0x7fcdb815c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:57.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 shutdown_connections 2024-04-25T15:20:57.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.213+0000 7fcdbebc5640 1 -- 172.21.15.146:0/4036516241 wait complete. 2024-04-25T15:20:57.216 INFO:tasks.ceph.mds.b.smithi177.stderr: -4690> 2024-04-25T15:19:50.737+0000 7fa569503640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:20:57.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:20:57.273 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-04-25T15:20:57.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.335+0000 7f3c5f172640 1 Processor -- start 2024-04-25T15:20:57.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.335+0000 7f3c5f172640 1 -- start start 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5f172640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3c58163c00 0x7f3c5806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5f172640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c580bb450 0x7f3c580d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5f172640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580d0de0 0x7f3c580b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5f172640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5806dea0 con 0x7f3c580bb450 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5f172640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3c580b5230 con 0x7f3c58163c00 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5f172640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3c5806c8c0 con 0x7f3c580d0de0 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c580bb450 0x7f3c580d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c580bb450 0x7f3c580d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55730/0 (socket says 172.21.15.146:55730) 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 -- 172.21.15.146:0/1904797054 learned_addr learned my addr 172.21.15.146:0/1904797054 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5d70f640 1 --2- 172.21.15.146:0/1904797054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580d0de0 0x7f3c580b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5cf0e640 1 --2- 172.21.15.146:0/1904797054 >> v2:172.21.15.177:3300/0 conn(0x7f3c58163c00 0x7f3c5806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 -- 172.21.15.146:0/1904797054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580d0de0 msgr2=0x7f3c580b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 --2- 172.21.15.146:0/1904797054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580d0de0 0x7f3c580b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 -- 172.21.15.146:0/1904797054 >> v2:172.21.15.177:3300/0 conn(0x7f3c58163c00 msgr2=0x7f3c5806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 --2- 172.21.15.146:0/1904797054 >> v2:172.21.15.177:3300/0 conn(0x7f3c58163c00 0x7f3c5806d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c57fff640 1 -- 172.21.15.146:0/1904797054 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5806cba0 con 0x7f3c580bb450 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.337+0000 7f3c5cf0e640 1 --2- 172.21.15.146:0/1904797054 >> v2:172.21.15.177:3300/0 conn(0x7f3c58163c00 0x7f3c5806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.338+0000 7f3c57fff640 1 --2- 172.21.15.146:0/1904797054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c580bb450 0x7f3c580d08a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3c5808e920 tx=0x7f3c4c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adf767b3e39dfd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.338+0000 7f3c577fe640 1 -- 172.21.15.146:0/1904797054 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c037070 con 0x7f3c580bb450 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.338+0000 7f3c577fe640 1 -- 172.21.15.146:0/1904797054 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c032440 con 0x7f3c580bb450 2024-04-25T15:20:57.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.338+0000 7f3c577fe640 1 -- 172.21.15.146:0/1904797054 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c032810 con 0x7f3c580bb450 2024-04-25T15:20:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 -- 172.21.15.146:0/1904797054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c580bb450 msgr2=0x7f3c580d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1904797054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c580bb450 0x7f3c580d08a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3c5808e920 tx=0x7f3c4c02a140 comp rx=0 tx=0).stop 2024-04-25T15:20:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 -- 172.21.15.146:0/1904797054 shutdown_connections 2024-04-25T15:20:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1904797054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580d0de0 0x7f3c580b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1904797054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c580bb450 0x7f3c580d08a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1904797054 >> v2:172.21.15.177:3300/0 conn(0x7f3c58163c00 0x7f3c5806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 -- 172.21.15.146:0/1904797054 >> 172.21.15.146:0/1904797054 conn(0x7f3c580b7970 msgr2=0x7f3c580b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 -- 172.21.15.146:0/1904797054 shutdown_connections 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.339+0000 7f3c5f172640 1 -- 172.21.15.146:0/1904797054 wait complete. 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.341+0000 7f3c5f172640 1 Processor -- start 2024-04-25T15:20:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.341+0000 7f3c5f172640 1 -- start start 2024-04-25T15:20:57.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5f172640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580bb450 0x7f3c58150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5f172640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c580d0de0 0x7f3c5814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5f172640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 0x7f3c580b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5f172640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5806c550 con 0x7f3c58163c00 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5f172640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5806d160 con 0x7f3c580d0de0 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5f172640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3c580bb860 con 0x7f3c580bb450 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 0x7f3c580b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 0x7f3c580b59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55734/0 (socket says 172.21.15.146:55734) 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 -- 172.21.15.146:0/1891827697 learned_addr learned my addr 172.21.15.146:0/1891827697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5cf0e640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580bb450 0x7f3c58150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c57fff640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c580d0de0 0x7f3c5814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 -- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580bb450 msgr2=0x7f3c58150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580bb450 0x7f3c58150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 -- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c580d0de0 msgr2=0x7f3c5814e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c580d0de0 0x7f3c5814e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:57.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.342+0000 7f3c5d70f640 1 -- 172.21.15.146:0/1891827697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c58151410 con 0x7f3c58163c00 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.343+0000 7f3c5d70f640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 0x7f3c580b59e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c50007d10 tx=0x7f3c5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.343+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50014420 con 0x7f3c58163c00 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.343+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c50019070 con 0x7f3c58163c00 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.343+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c58151920 con 0x7f3c58163c00 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.343+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50014800 con 0x7f3c58163c00 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.343+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5814f000 con 0x7f3c58163c00 2024-04-25T15:20:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.344+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5806a0e0 con 0x7f3c58163c00 2024-04-25T15:20:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.344+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3c500149a0 con 0x7f3c58163c00 2024-04-25T15:20:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.345+0000 7f3c557fa640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3c440744b0 0x7f3c440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.345+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6800+0+0 (secure 0 0 0) 0x7f3c50095c60 con 0x7f3c58163c00 2024-04-25T15:20:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.345+0000 7f3c5cf0e640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3c440744b0 0x7f3c440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.345+0000 7f3c5cf0e640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3c440744b0 0x7f3c440769c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3c580b5130 tx=0x7f3c48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:57.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.350+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c5005f9f0 con 0x7f3c58163c00 2024-04-25T15:20:57.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:57.563+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3c581516b0 con 0x7f3c58163c00 2024-04-25T15:20:58.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.211+0000 7f3c557fa640 1 -- 172.21.15.146:0/1891827697 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f3c50063cf0 con 0x7f3c58163c00 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.213+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3c440744b0 msgr2=0x7f3c440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.213+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3c440744b0 0x7f3c440769c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3c580b5130 tx=0x7f3c48008040 comp rx=0 tx=0).stop 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 msgr2=0x7f3c580b59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 0x7f3c580b59e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c50007d10 tx=0x7f3c5000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 shutdown_connections 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3c440744b0 0x7f3c440769c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c58163c00 0x7f3c580b59e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c580d0de0 0x7f3c5814e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 --2- 172.21.15.146:0/1891827697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3c580bb450 0x7f3c58150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 >> 172.21.15.146:0/1891827697 conn(0x7f3c580b7970 msgr2=0x7f3c580b8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 shutdown_connections 2024-04-25T15:20:58.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.214+0000 7f3c5f172640 1 -- 172.21.15.146:0/1891827697 wait complete. 2024-04-25T15:20:58.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:20:58.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.344+0000 7fcb80f38640 1 Processor -- start 2024-04-25T15:20:58.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.344+0000 7fcb80f38640 1 -- start start 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb80f38640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb7c0d2840 0x7fcb7c0d2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb80f38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb80f38640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcb7c06cde0 0x7fcb7c06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb80f38640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7c0cef00 con 0x7fcb7c0d2840 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb80f38640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7c0cff80 con 0x7fcb7c06cde0 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb80f38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7c0cf1c0 con 0x7fcb7c0ce1f0 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb7c0d2840 0x7fcb7c0d2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb7c0d2840 0x7fcb7c0d2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55738/0 (socket says 172.21.15.146:55738) 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 -- 172.21.15.146:0/558146374 learned_addr learned my addr 172.21.15.146:0/558146374 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb79d9b640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7ad9d640 1 --2- 172.21.15.146:0/558146374 >> v2:172.21.15.177:3300/0 conn(0x7fcb7c06cde0 0x7fcb7c06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 -- 172.21.15.146:0/558146374 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 msgr2=0x7fcb7c0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 -- 172.21.15.146:0/558146374 >> v2:172.21.15.177:3300/0 conn(0x7fcb7c06cde0 msgr2=0x7fcb7c06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 --2- 172.21.15.146:0/558146374 >> v2:172.21.15.177:3300/0 conn(0x7fcb7c06cde0 0x7fcb7c06d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb7a59c640 1 -- 172.21.15.146:0/558146374 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c0cf4a0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.346+0000 7fcb79d9b640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.347+0000 7fcb7a59c640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0d2c50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b5320 tx=0x7fcb6c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cce5a97ac02b99c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.347+0000 7fcb7959a640 1 -- 172.21.15.146:0/558146374 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c036070 con 0x7fcb7c0d2840 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.347+0000 7fcb7959a640 1 -- 172.21.15.146:0/558146374 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb6c031600 con 0x7fcb7c0d2840 2024-04-25T15:20:58.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.347+0000 7fcb7959a640 1 -- 172.21.15.146:0/558146374 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c0319d0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 -- 172.21.15.146:0/558146374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 msgr2=0x7fcb7c0d2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:58.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0d2c50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b5320 tx=0x7fcb6c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 -- 172.21.15.146:0/558146374 shutdown_connections 2024-04-25T15:20:58.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 --2- 172.21.15.146:0/558146374 >> v2:172.21.15.177:3300/0 conn(0x7fcb7c06cde0 0x7fcb7c06d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 --2- 172.21.15.146:0/558146374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0d2c50 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 -- 172.21.15.146:0/558146374 >> 172.21.15.146:0/558146374 conn(0x7fcb7c0b7460 msgr2=0x7fcb7c0b7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 -- 172.21.15.146:0/558146374 shutdown_connections 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.348+0000 7fcb80f38640 1 -- 172.21.15.146:0/558146374 wait complete. 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.350+0000 7fcb80f38640 1 Processor -- start 2024-04-25T15:20:58.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.350+0000 7fcb80f38640 1 -- start start 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb80f38640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb7c06cde0 0x7fcb7c0b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb80f38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb80f38640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb80f38640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c0ceb40 con 0x7fcb7c0d2840 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb80f38640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c0cfa60 con 0x7fcb7c06cde0 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb80f38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7c0cecc0 con 0x7fcb7c0ce1f0 2024-04-25T15:20:58.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb7ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb7ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55750/0 (socket says 172.21.15.146:55750) 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb7ad9d640 1 -- 172.21.15.146:0/1007217646 learned_addr learned my addr 172.21.15.146:0/1007217646 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb79d9b640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.351+0000 7fcb7a59c640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb7c06cde0 0x7fcb7c0b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb7ad9d640 1 -- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 msgr2=0x7fcb7c0c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb7ad9d640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb7ad9d640 1 -- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb7c06cde0 msgr2=0x7fcb7c0b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb7ad9d640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb7c06cde0 0x7fcb7c0b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb7ad9d640 1 -- 172.21.15.146:0/1007217646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c0c92c0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb7ad9d640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0c7ab0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c480 tx=0x7fcb7400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74015070 con 0x7fcb7c0d2840 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb74004220 con 0x7fcb7c0d2840 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7c0c6a10 con 0x7fcb7c0d2840 2024-04-25T15:20:58.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74003610 con 0x7fcb7c0d2840 2024-04-25T15:20:58.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.352+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7c0c6fd0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.353+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb7c06a0e0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.354+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb740037b0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.354+0000 7fcb6affd640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcb5c0744b0 0x7fcb5c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:58.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.354+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6800+0+0 (secure 0 0 0) 0x7fcb74095230 con 0x7fcb7c0d2840 2024-04-25T15:20:58.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.354+0000 7fcb7a59c640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcb5c0744b0 0x7fcb5c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:58.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.355+0000 7fcb7a59c640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcb5c0744b0 0x7fcb5c0769c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c033000 tx=0x7fcb6c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:58.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.359+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb7c0c92c0 con 0x7fcb7c0d2840 2024-04-25T15:20:58.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:58.538+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcb7c0c6760 con 0x7fcb7c0d2840 2024-04-25T15:20:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.212+0000 7fcb6affd640 1 -- 172.21.15.146:0/1007217646 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7fcb7405f110 con 0x7fcb7c0d2840 2024-04-25T15:20:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.217+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcb5c0744b0 msgr2=0x7fcb5c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.217+0000 7fcb80f38640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcb5c0744b0 0x7fcb5c0769c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c033000 tx=0x7fcb6c009000 comp rx=0 tx=0).stop 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.218+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 msgr2=0x7fcb7c0c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.218+0000 7fcb80f38640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0c7ab0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c480 tx=0x7fcb7400c940 comp rx=0 tx=0).stop 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 shutdown_connections 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcb5c0744b0 0x7fcb5c0769c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb7c0d2840 0x7fcb7c0c7ab0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcb7c0ce1f0 0x7fcb7c0c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 --2- 172.21.15.146:0/1007217646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb7c06cde0 0x7fcb7c0b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 >> 172.21.15.146:0/1007217646 conn(0x7fcb7c0b7460 msgr2=0x7fcb7c15f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 shutdown_connections 2024-04-25T15:20:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.219+0000 7fcb80f38640 1 -- 172.21.15.146:0/1007217646 wait complete. 2024-04-25T15:20:59.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:20:59.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.352+0000 7f568eb14640 1 Processor -- start 2024-04-25T15:20:59.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.352+0000 7f568eb14640 1 -- start start 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.353+0000 7f568eb14640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f56880b8990 0x7f568815f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.353+0000 7f568eb14640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568808cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568eb14640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f568808d130 0x7f56880d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568eb14640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f56880ce550 con 0x7f568808d130 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568eb14640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f56880ce7f0 con 0x7f56880b8990 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568eb14640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f56880cead0 con 0x7f56880cf5b0 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f568808d130 0x7f56880d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f568808d130 0x7f56880d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55764/0 (socket says 172.21.15.146:55764) 2024-04-25T15:20:59.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 -- 172.21.15.146:0/1037278484 learned_addr learned my addr 172.21.15.146:0/1037278484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568c8b0640 1 --2- 172.21.15.146:0/1037278484 >> v2:172.21.15.177:3300/0 conn(0x7f56880b8990 0x7f568815f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 -- 172.21.15.146:0/1037278484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 msgr2=0x7f568808cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 --2- 172.21.15.146:0/1037278484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568808cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 -- 172.21.15.146:0/1037278484 >> v2:172.21.15.177:3300/0 conn(0x7f56880b8990 msgr2=0x7f568815f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 --2- 172.21.15.146:0/1037278484 >> v2:172.21.15.177:3300/0 conn(0x7f56880b8990 0x7f568815f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568d0b1640 1 -- 172.21.15.146:0/1037278484 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56880cedb0 con 0x7f568808d130 2024-04-25T15:20:59.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.354+0000 7f568c8b0640 1 --2- 172.21.15.146:0/1037278484 >> v2:172.21.15.177:3300/0 conn(0x7f56880b8990 0x7f568815f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:59.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.355+0000 7f568d0b1640 1 --2- 172.21.15.146:0/1037278484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f568808d130 0x7f56880d13d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56880b98b0 tx=0x7f568002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af4b58803a27535 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:59.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.355+0000 7f56877fe640 1 -- 172.21.15.146:0/1037278484 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680036070 con 0x7f568808d130 2024-04-25T15:20:59.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.355+0000 7f56877fe640 1 -- 172.21.15.146:0/1037278484 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5680031600 con 0x7f568808d130 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.355+0000 7f56877fe640 1 -- 172.21.15.146:0/1037278484 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800319d0 con 0x7f568808d130 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 -- 172.21.15.146:0/1037278484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f568808d130 msgr2=0x7f56880d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 --2- 172.21.15.146:0/1037278484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f568808d130 0x7f56880d13d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56880b98b0 tx=0x7f568002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 -- 172.21.15.146:0/1037278484 shutdown_connections 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 --2- 172.21.15.146:0/1037278484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f568808d130 0x7f56880d13d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 --2- 172.21.15.146:0/1037278484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568808cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 --2- 172.21.15.146:0/1037278484 >> v2:172.21.15.177:3300/0 conn(0x7f56880b8990 0x7f568815f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 -- 172.21.15.146:0/1037278484 >> 172.21.15.146:0/1037278484 conn(0x7f56880b52f0 msgr2=0x7f56880b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 -- 172.21.15.146:0/1037278484 shutdown_connections 2024-04-25T15:20:59.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.356+0000 7f568eb14640 1 -- 172.21.15.146:0/1037278484 wait complete. 2024-04-25T15:20:59.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.357+0000 7f568eb14640 1 Processor -- start 2024-04-25T15:20:59.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.357+0000 7f568eb14640 1 -- start start 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568eb14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f568808d130 0x7f56881e9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568eb14640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 0x7f56880a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568eb14640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568815ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568eb14640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f56880b90f0 con 0x7f56880b8990 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568eb14640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f568808d540 con 0x7f568808d130 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568eb14640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f56880cf9c0 con 0x7f56880cf5b0 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 0x7f56880a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 0x7f56880a0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55772/0 (socket says 172.21.15.146:55772) 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 -- 172.21.15.146:0/3334736824 learned_addr learned my addr 172.21.15.146:0/3334736824 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:20:59.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568c8b0640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f568808d130 0x7f56881e9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568d0b1640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568815ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 -- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 msgr2=0x7f568815ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568815ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 -- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f568808d130 msgr2=0x7f56881e9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f568808d130 0x7f56881e9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f5687fff640 1 -- 172.21.15.146:0/3334736824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56881e8e00 con 0x7f56880b8990 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.359+0000 7f568d0b1640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568815ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.360+0000 7f5687fff640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 0x7f56880a0f70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f567c00b440 tx=0x7f567c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.360+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c015020 con 0x7f56880b8990 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.360+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c011830 con 0x7f56880b8990 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.360+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56881e90e0 con 0x7f56880b8990 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.360+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c011c20 con 0x7f56880b8990 2024-04-25T15:20:59.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.360+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56881e96a0 con 0x7f56880b8990 2024-04-25T15:20:59.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.361+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568806a0e0 con 0x7f56880b8990 2024-04-25T15:20:59.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.362+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f567c018020 con 0x7f56880b8990 2024-04-25T15:20:59.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.362+0000 7f56857fa640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56740744e0 0x7f56740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:20:59.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.362+0000 7f568c8b0640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56740744e0 0x7f56740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:20:59.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.362+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6362+0+0 (secure 0 0 0) 0x7f567c095410 con 0x7f56880b8990 2024-04-25T15:20:59.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.363+0000 7f568c8b0640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56740744e0 0x7f56740769f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5688074790 tx=0x7f5678008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:20:59.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.367+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f567c01f020 con 0x7f56880b8990 2024-04-25T15:20:59.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:20:59.547+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f56880d35a0 con 0x7f56880b8990 2024-04-25T15:21:00.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.214+0000 7f56857fa640 1 -- 172.21.15.146:0/3334736824 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7f567c05f2c0 con 0x7f56880b8990 2024-04-25T15:21:00.215 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' removed 2024-04-25T15:21:00.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.221+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56740744e0 msgr2=0x7f56740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.221+0000 7f568eb14640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56740744e0 0x7f56740769f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5688074790 tx=0x7f5678008040 comp rx=0 tx=0).stop 2024-04-25T15:21:00.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.221+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 msgr2=0x7f56880a0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.221+0000 7f568eb14640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 0x7f56880a0f70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f567c00b440 tx=0x7f567c00b900 comp rx=0 tx=0).stop 2024-04-25T15:21:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 shutdown_connections 2024-04-25T15:21:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f56740744e0 0x7f56740769f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56880cf5b0 0x7f568815ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56880b8990 0x7f56880a0f70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 --2- 172.21.15.146:0/3334736824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f568808d130 0x7f56881e9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 >> 172.21.15.146:0/3334736824 conn(0x7f56880b52f0 msgr2=0x7f56880cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:00.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 shutdown_connections 2024-04-25T15:21:00.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.222+0000 7f568eb14640 1 -- 172.21.15.146:0/3334736824 wait complete. 2024-04-25T15:21:00.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T15:21:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.351+0000 7f7301d7d640 1 Processor -- start 2024-04-25T15:21:00.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.351+0000 7f7301d7d640 1 -- start start 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.352+0000 7f7301d7d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f72fc146140 0x7f72fc1601a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.352+0000 7f7301d7d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72fc0cd380 con 0x7f72fc06d4f0 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.352+0000 7f7301d7d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72fc0cd660 con 0x7f72fc145bd0 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.352+0000 7f7301d7d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f72fc0cd940 con 0x7f72fc146140 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72faffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f72fc06d4f0 0x7f72fc15ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45952/0 (socket says 172.21.15.146:45952) 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72faffd640 1 -- 172.21.15.146:0/1915500773 learned_addr learned my addr 172.21.15.146:0/1915500773 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 189543097 0 0) 0x7f72fc0cd380 con 0x7f72fc06d4f0 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fbfff640 1 --2- 172.21.15.146:0/1915500773 >> v2:172.21.15.177:3300/0 conn(0x7f72fc146140 0x7f72fc1601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72e4004630 con 0x7f72fc06d4f0 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3095340437 0 0) 0x7f72e4004630 con 0x7f72fc06d4f0 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 >> v2:172.21.15.177:3300/0 conn(0x7f72fc146140 msgr2=0x7f72fc1601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 --2- 172.21.15.146:0/1915500773 >> v2:172.21.15.177:3300/0 conn(0x7f72fc146140 0x7f72fc1601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 >> v1:172.21.15.177:6789/0 conn(0x7f72fc145bd0 legacy=0x7f72fc06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72fc0cdc20 con 0x7f72fc06d4f0 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f72f0001670 con 0x7f72fc06d4f0 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 >> v1:172.21.15.146:6789/0 conn(0x7f72fc06d4f0 legacy=0x7f72fc15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e4004ac0 0x7f72e4004f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72e4005910 0x7f72e4005d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72e40062d0 0x7f72e400cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc0cd940 con 0x7f72e40062d0 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72e400d270 con 0x7f72e4005910 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f72e400d470 con 0x7f72e4004ac0 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fbfff640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72e40062d0 0x7f72e400cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72faffd640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72e4005910 0x7f72e4005d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.353+0000 7f72fb7fe640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e4004ac0 0x7f72e4004f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 -- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e4004ac0 msgr2=0x7f72e4004f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e4004ac0 0x7f72e4004f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 -- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72e4005910 msgr2=0x7f72e4005d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72e4005910 0x7f72e4005d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 -- 172.21.15.146:0/1915500773 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e4004630 con 0x7f72e40062d0 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 -- 172.21.15.146:0/1915500773 reap_dead start 2024-04-25T15:21:00.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.354+0000 7f72fbfff640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72e40062d0 0x7f72e400cad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009f90 tx=0x7f72e4006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08394d29b3422b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4009c10 con 0x7f72e40062d0 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f4003b90 con 0x7f72e40062d0 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f72fa7fc640 1 -- 172.21.15.146:0/1915500773 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4004020 con 0x7f72e40062d0 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 -- 172.21.15.146:0/1915500773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72e40062d0 msgr2=0x7f72e400cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72e40062d0 0x7f72e400cad0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009f90 tx=0x7f72e4006940 comp rx=0 tx=0).stop 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 -- 172.21.15.146:0/1915500773 shutdown_connections 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 --2- 172.21.15.146:0/1915500773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72e40062d0 0x7f72e400cad0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 -- 172.21.15.146:0/1915500773 >> 172.21.15.146:0/1915500773 conn(0x7f72fc0ce470 msgr2=0x7f72fc0ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 -- 172.21.15.146:0/1915500773 shutdown_connections 2024-04-25T15:21:00.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.355+0000 7f7301d7d640 1 -- 172.21.15.146:0/1915500773 wait complete. 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.357+0000 7f7301d7d640 1 Processor -- start 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.357+0000 7f7301d7d640 1 -- start start 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e40062d0 0x7f72fc144230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 0x7f72fc09c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72fc1f74a0 0x7f72fc1f78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc0cd120 con 0x7f72fc09bc20 2024-04-25T15:21:00.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc0ccfa0 con 0x7f72fc1f74a0 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f72fc071050 con 0x7f72e40062d0 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 0x7f72fc09c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 0x7f72fc09c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55790/0 (socket says 172.21.15.146:55790) 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 -- 172.21.15.146:0/2654911614 learned_addr learned my addr 172.21.15.146:0/2654911614 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72fbfff640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72fc1f74a0 0x7f72fc1f78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72fb7fe640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e40062d0 0x7f72fc144230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 -- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e40062d0 msgr2=0x7f72fc144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e40062d0 0x7f72fc144230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 -- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72fc1f74a0 msgr2=0x7f72fc1f78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72fc1f74a0 0x7f72fc1f78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 -- 172.21.15.146:0/2654911614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72fc1f7df0 con 0x7f72fc09bc20 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72faffd640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 0x7f72fc09c030 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f72f0004f40 tx=0x7f72f00074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f000f7b0 con 0x7f72fc09bc20 2024-04-25T15:21:00.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f0018070 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72fc09c950 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f000fb90 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.359+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72fc09cf10 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.360+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72fc09f790 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.361+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f72f000fd30 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.361+0000 7f72e3fff640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72e80744e0 0x7f72e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.361+0000 7f72fb7fe640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72e80744e0 0x7f72e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.361+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7f72f00952a0 con 0x7f72fc09bc20 2024-04-25T15:21:00.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.362+0000 7f72fb7fe640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72e80744e0 0x7f72e80769f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f72ec000c00 tx=0x7f72ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:00.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.366+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72fc09f790 con 0x7f72fc09bc20 2024-04-25T15:21:00.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:00.542+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f72fc06a0e0 con 0x7f72fc09bc20 2024-04-25T15:21:01.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.225+0000 7f72e3fff640 1 -- 172.21.15.146:0/2654911614 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7f72fc1f7df0 con 0x7f72fc09bc20 2024-04-25T15:21:01.226 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data_ec' removed 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.228+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72e80744e0 msgr2=0x7f72e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.228+0000 7f7301d7d640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72e80744e0 0x7f72e80769f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f72ec000c00 tx=0x7f72ec009040 comp rx=0 tx=0).stop 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.228+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 msgr2=0x7f72fc09c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.228+0000 7f7301d7d640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 0x7f72fc09c030 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f72f0004f40 tx=0x7f72f00074f0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 shutdown_connections 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72e80744e0 0x7f72e80769f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72fc1f74a0 0x7f72fc1f78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc09bc20 0x7f72fc09c030 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 --2- 172.21.15.146:0/2654911614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72e40062d0 0x7f72fc144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 >> 172.21.15.146:0/2654911614 conn(0x7f72fc0ce470 msgr2=0x7f72fc145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 shutdown_connections 2024-04-25T15:21:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.229+0000 7f7301d7d640 1 -- 172.21.15.146:0/2654911614 wait complete. 2024-04-25T15:21:01.245 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:21:01.245 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:21:01.246 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:21:01.246 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:21:01.246 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:21:01.246 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:21:01.246 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:21:01.246 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:21:01.246 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:21:01.247 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:21:01.247 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:21:01.247 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:21:01.247 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:21:01.247 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:21:01.247 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:21:01.247 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:21:01.256 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:21:01.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:21:01.257 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:21:01.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:21:01.260 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:21:01.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:21:01.262 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:21:01.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:21:01.299 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:21:01.299 DEBUG:teuthology.parallel:result is None 2024-04-25T15:21:01.300 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:21:01.300 DEBUG:teuthology.parallel:result is None 2024-04-25T15:21:01.304 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:21:01.304 DEBUG:teuthology.parallel:result is None 2024-04-25T15:21:01.305 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:21:01.306 DEBUG:teuthology.parallel:result is None 2024-04-25T15:21:01.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:21:01.386 INFO:tasks.ceph.mds.a.smithi146.stdout:starting mds.a at 2024-04-25T15:21:01.387 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-04-25T15:21:01.398 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-04-25T15:21:01.398 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-04-25T15:21:01.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.422+0000 7ffba0757640 1 Processor -- start 2024-04-25T15:21:01.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.422+0000 7ffba0757640 1 -- start start 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.423+0000 7ffba0757640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d4f60 0x7ffb980d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffba0757640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffb980d0020 0x7ffb980d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffba0757640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb98163d80 0x7ffb98164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffba0757640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9806c550 con 0x7ffb98163d80 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffba0757640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9806c830 con 0x7ffb980d0020 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffba0757640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffb980b9200 con 0x7ffb980d4f60 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb98163d80 0x7ffb98164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb98163d80 0x7ffb98164190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48696/0 (socket says 172.21.15.146:48696) 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/913978015 learned_addr learned my addr 172.21.15.146:0/913978015 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9dcf2640 1 --2- 172.21.15.146:0/913978015 >> v2:172.21.15.177:3300/0 conn(0x7ffb980d0020 0x7ffb980d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9e4f3640 1 --2- 172.21.15.146:0/913978015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d4f60 0x7ffb980d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/913978015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d4f60 msgr2=0x7ffb980d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:01.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 --2- 172.21.15.146:0/913978015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d4f60 0x7ffb980d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/913978015 >> v2:172.21.15.177:3300/0 conn(0x7ffb980d0020 msgr2=0x7ffb980d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:01.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 --2- 172.21.15.146:0/913978015 >> v2:172.21.15.177:3300/0 conn(0x7ffb980d0020 0x7ffb980d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.424+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/913978015 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb980d0970 con 0x7ffb98163d80 2024-04-25T15:21:01.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.425+0000 7ffb9ecf4640 1 --2- 172.21.15.146:0/913978015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb98164190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ffb9806f9e0 tx=0x7ffb9002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18ca0dae8f9a5ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:01.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.425+0000 7ffb9d4f1640 1 -- 172.21.15.146:0/913978015 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90036070 con 0x7ffb98163d80 2024-04-25T15:21:01.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.425+0000 7ffb9d4f1640 1 -- 172.21.15.146:0/913978015 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb90031600 con 0x7ffb98163d80 2024-04-25T15:21:01.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.425+0000 7ffb9d4f1640 1 -- 172.21.15.146:0/913978015 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb900319d0 con 0x7ffb98163d80 2024-04-25T15:21:01.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 -- 172.21.15.146:0/913978015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 msgr2=0x7ffb98164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 --2- 172.21.15.146:0/913978015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb98164190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ffb9806f9e0 tx=0x7ffb9002a570 comp rx=0 tx=0).stop 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 -- 172.21.15.146:0/913978015 shutdown_connections 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 --2- 172.21.15.146:0/913978015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb98164190 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 --2- 172.21.15.146:0/913978015 >> v2:172.21.15.177:3300/0 conn(0x7ffb980d0020 0x7ffb980d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 --2- 172.21.15.146:0/913978015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d4f60 0x7ffb980d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 -- 172.21.15.146:0/913978015 >> 172.21.15.146:0/913978015 conn(0x7ffb980d2450 msgr2=0x7ffb980d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 -- 172.21.15.146:0/913978015 shutdown_connections 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.426+0000 7ffba0757640 1 -- 172.21.15.146:0/913978015 wait complete. 2024-04-25T15:21:01.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.427+0000 7ffba0757640 1 Processor -- start 2024-04-25T15:21:01.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.428+0000 7ffba0757640 1 -- start start 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffba0757640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d0020 0x7ffb9815e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffba0757640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb980d4f60 0x7ffb98157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffba0757640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb981f9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffba0757640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffb980d15f0 con 0x7ffb98163d80 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffba0757640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb980d1070 con 0x7ffb980d4f60 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffba0757640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffb980b5150 con 0x7ffb980d0020 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb981f9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb981f9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48710/0 (socket says 172.21.15.146:48710) 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/4102562938 learned_addr learned my addr 172.21.15.146:0/4102562938 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9e4f3640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d0020 0x7ffb9815e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9dcf2640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb980d4f60 0x7ffb98157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d0020 msgr2=0x7ffb9815e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d0020 0x7ffb9815e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb980d4f60 msgr2=0x7ffb98157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb980d4f60 0x7ffb98157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 -- 172.21.15.146:0/4102562938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9815ed00 con 0x7ffb98163d80 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9e4f3640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d0020 0x7ffb9815e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9dcf2640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb980d4f60 0x7ffb98157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.429+0000 7ffb9ecf4640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb981f9fa0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009f40 tx=0x7ffb90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.430+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90036050 con 0x7ffb98163d80 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.430+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb90042070 con 0x7ffb98163d80 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.430+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90009510 con 0x7ffb98163d80 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.430+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb9815efe0 con 0x7ffb98163d80 2024-04-25T15:21:01.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.430+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb9815f5a0 con 0x7ffb98163d80 2024-04-25T15:21:01.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.431+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb9806a0e0 con 0x7ffb98163d80 2024-04-25T15:21:01.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.431+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffb90007c70 con 0x7ffb98163d80 2024-04-25T15:21:01.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.431+0000 7ffb8affd640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb7c0744b0 0x7ffb7c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:01.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.432+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7ffb900b5e20 con 0x7ffb98163d80 2024-04-25T15:21:01.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.432+0000 7ffb9e4f3640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb7c0744b0 0x7ffb7c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:01.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.432+0000 7ffb9e4f3640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb7c0744b0 0x7ffb7c0769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb980b4880 tx=0x7ffb94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.436+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb980d0970 con 0x7ffb98163d80 2024-04-25T15:21:01.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:01.621+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ffb9806cfb0 con 0x7ffb98163d80 2024-04-25T15:21:02.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.236+0000 7ffb8affd640 1 -- 172.21.15.146:0/4102562938 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7ffb900807b0 con 0x7ffb98163d80 2024-04-25T15:21:02.237 INFO:teuthology.orchestra.run.smithi146.stderr: removed all blocklist entries 2024-04-25T15:21:02.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb7c0744b0 msgr2=0x7ffb7c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb7c0744b0 0x7ffb7c0769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb980b4880 tx=0x7ffb94008040 comp rx=0 tx=0).stop 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 msgr2=0x7ffb981f9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb981f9fa0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009f40 tx=0x7ffb90004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 shutdown_connections 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb7c0744b0 0x7ffb7c0769c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb98163d80 0x7ffb981f9fa0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb980d4f60 0x7ffb98157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 --2- 172.21.15.146:0/4102562938 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb980d0020 0x7ffb9815e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 >> 172.21.15.146:0/4102562938 conn(0x7ffb980d2450 msgr2=0x7ffb9806d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 shutdown_connections 2024-04-25T15:21:02.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.239+0000 7ffba0757640 1 -- 172.21.15.146:0/4102562938 wait complete. 2024-04-25T15:21:02.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:21:02.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.358+0000 7fbe4ea92640 1 Processor -- start 2024-04-25T15:21:02.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.358+0000 7fbe4ea92640 1 -- start start 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4ea92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe480b7350 0x7fbe480715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4ea92640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbe48070360 0x7fbe48169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4ea92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe48071bc0 con 0x7fbe48160000 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4ea92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbe480d1b00 con 0x7fbe480b7350 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4ea92640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbe480d1de0 con 0x7fbe48070360 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4c82e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe480b7350 0x7fbe480715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe47fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbe48160000 0x7fbe4806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41694/0 (socket says 172.21.15.146:41694) 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe47fff640 1 -- 172.21.15.146:0/3015177303 learned_addr learned my addr 172.21.15.146:0/3015177303 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4d02f640 1 --2- 172.21.15.146:0/3015177303 >> v2:172.21.15.177:3300/0 conn(0x7fbe48070360 0x7fbe48169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4c82e640 1 -- 172.21.15.146:0/3015177303 >> v2:172.21.15.177:3300/0 conn(0x7fbe48070360 msgr2=0x7fbe48169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:02.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4c82e640 1 --2- 172.21.15.146:0/3015177303 >> v2:172.21.15.177:3300/0 conn(0x7fbe48070360 0x7fbe48169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4c82e640 1 -- 172.21.15.146:0/3015177303 >> v1:172.21.15.146:6789/0 conn(0x7fbe48160000 legacy=0x7fbe4806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe4c82e640 1 -- 172.21.15.146:0/3015177303 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe480d20c0 con 0x7fbe480b7350 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.360+0000 7fbe477fe640 1 -- 172.21.15.146:0/3015177303 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184677748 0 0) 0x7fbe48071bc0 con 0x7fbe48160000 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.361+0000 7fbe4c82e640 1 --2- 172.21.15.146:0/3015177303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe480b7350 0x7fbe480715e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbe480cf880 tx=0x7fbe3002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b1a0bbf69afcf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.361+0000 7fbe477fe640 1 -- 172.21.15.146:0/3015177303 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe30037070 con 0x7fbe480b7350 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.361+0000 7fbe477fe640 1 -- 172.21.15.146:0/3015177303 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe30032440 con 0x7fbe480b7350 2024-04-25T15:21:02.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.361+0000 7fbe477fe640 1 -- 172.21.15.146:0/3015177303 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe30032810 con 0x7fbe480b7350 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 -- 172.21.15.146:0/3015177303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe480b7350 msgr2=0x7fbe480715e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/3015177303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe480b7350 0x7fbe480715e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbe480cf880 tx=0x7fbe3002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 -- 172.21.15.146:0/3015177303 shutdown_connections 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/3015177303 >> v2:172.21.15.177:3300/0 conn(0x7fbe48070360 0x7fbe48169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/3015177303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe480b7350 0x7fbe480715e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 -- 172.21.15.146:0/3015177303 >> 172.21.15.146:0/3015177303 conn(0x7fbe4809b2b0 msgr2=0x7fbe480b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 -- 172.21.15.146:0/3015177303 shutdown_connections 2024-04-25T15:21:02.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.362+0000 7fbe4ea92640 1 -- 172.21.15.146:0/3015177303 wait complete. 2024-04-25T15:21:02.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.364+0000 7fbe4ea92640 1 Processor -- start 2024-04-25T15:21:02.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.364+0000 7fbe4ea92640 1 -- start start 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4ea92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe480b7350 0x7fbe4815e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4ea92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 0x7fbe4809eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4ea92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe4809f410 0x7fbe480c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4ea92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbe48070980 con 0x7fbe48160000 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4ea92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbe480cf4b0 con 0x7fbe480b7350 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4ea92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbe48160410 con 0x7fbe4809f410 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 0x7fbe4809eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 0x7fbe4809eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48720/0 (socket says 172.21.15.146:48720) 2024-04-25T15:21:02.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 -- 172.21.15.146:0/2666423846 learned_addr learned my addr 172.21.15.146:0/2666423846 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4d02f640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe4809f410 0x7fbe480c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe4c82e640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe480b7350 0x7fbe4815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 -- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe4809f410 msgr2=0x7fbe480c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe4809f410 0x7fbe480c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 -- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe480b7350 msgr2=0x7fbe4815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe480b7350 0x7fbe4815e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.365+0000 7fbe47fff640 1 -- 172.21.15.146:0/2666423846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4809db50 con 0x7fbe48160000 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.366+0000 7fbe47fff640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 0x7fbe4809eed0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800c7f0 tx=0x7fbe3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.366+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe38018070 con 0x7fbe48160000 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.366+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe3800a640 con 0x7fbe48160000 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.366+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4809fc00 con 0x7fbe48160000 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.366+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800aa30 con 0x7fbe48160000 2024-04-25T15:21:02.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.366+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe4809c2d0 con 0x7fbe48160000 2024-04-25T15:21:02.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.367+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe4806a0e0 con 0x7fbe48160000 2024-04-25T15:21:02.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.368+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbe3801b070 con 0x7fbe48160000 2024-04-25T15:21:02.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.368+0000 7fbe457fa640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe280744e0 0x7fbe280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.368+0000 7fbe4c82e640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe280744e0 0x7fbe280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.368+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fbe380945d0 con 0x7fbe48160000 2024-04-25T15:21:02.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.369+0000 7fbe4c82e640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe280744e0 0x7fbe280769f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe30034000 tx=0x7fbe3002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:02.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.373+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe3805efb0 con 0x7fbe48160000 2024-04-25T15:21:02.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.617+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fbe4809f990 con 0x7fbe48160000 2024-04-25T15:21:02.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.617+0000 7fbe457fa640 1 -- 172.21.15.146:0/2666423846 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fbe3805e8b0 con 0x7fbe48160000 2024-04-25T15:21:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:02.618 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_dump": [ 2024-04-25T15:21:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.a", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpm/Ux1DhAA3DlkdKegZpZxlaZPFjodBw==", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.b", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmR1PoGhAAw/gk2lg0yvzUcOUZvwo6jg==", 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.c", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmtDPsFRAAh1HLdvyYLR78fokIKaekrA==", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.d", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpm3EVMIhAAddbxnaiTto+gXb0+LGbRKA==", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.0", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBcdCpmkFKVABAA+xGQMOacMJi18fh2p3rJFA==", 2024-04-25T15:21:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.1", 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBedCpm78w0OxAAGf4e1jZ0aAxnX/HBagE3yg==", 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.2", 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBhdCpm2VCWORAAUEOB2XJBAbm1STvjkbjFNw==", 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.3", 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBkdCpmfHmAOBAApJr+ddhMkB3As5PU2KALaQ==", 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.4", 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBqdCpmtoKbORAA63fHSceN+QJCRCuFLZKlqA==", 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.5", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBtdCpmIkKAOBAAN4te1g8Ioao/xQtjvw4vSg==", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.6", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBwdCpm90+YNhAApJ3X6R6BJC2VXiJ+RCl+TA==", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.7", 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBzdCpmACtMNxAAFvU4+bypsr14tXk8suRhEQ==", 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.0", 2024-04-25T15:21:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmlczBJhAApZj6jMADYxBHeBvhaprLDg==", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.1", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmy/P3LRAAyZkQ/mpIZ9/H8m7/scVDKA==", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:21:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.2", 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmhV1cKhAAdJQ+5g5RjIlilhKuxLcyiA==", 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.3", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmR6yLMRAAhFRT/TgCAnREAuUr53J1BA==", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.admin", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBXdCpmiIoCLRAAnMVBi4QtXLwcDZ/1klNNcg==", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow *", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:21:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.bootstrap-mds", 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQB5dCpmGGP7MxAAsJZS8wbNZ6hE2sTKSjqVQg==", 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile bootstrap-mds" 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.bootstrap-mgr", 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQB5dCpm1mr7MxAAUIuU+FkpzlJoTxdrZj7tjw==", 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.bootstrap-osd", 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQB5dCpmt3H7MxAAz3T2xxKwKlAxJa+8pJUbsA==", 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile bootstrap-osd" 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.bootstrap-rbd", 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQB5dCpmWHj7MxAAS/r6DKdsfvTHUad3fPjnnQ==", 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQB5dCpmgoD7MxAAZJjzr+pDlhQgvYyOR3JVTA==", 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.bootstrap-rgw", 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQB5dCpmpIf7MxAA41FefrC5J5clV6pTK0RmNw==", 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mgr.x", 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmiMrSChAA/AOecro7QL+ypBbUDiwShQ==", 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow *", 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile mgr", 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mgr.y", 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmmk8XBhAASX5Dc72zChEYixIh/iiawg==", 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow *", 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile mgr", 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-25T15:21:02.634 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe280744e0 msgr2=0x7fbe280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe280744e0 0x7fbe280769f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe30034000 tx=0x7fbe3002e000 comp rx=0 tx=0).stop 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 msgr2=0x7fbe4809eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 0x7fbe4809eed0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800c7f0 tx=0x7fbe3800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 shutdown_connections 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbe280744e0 0x7fbe280769f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe4809f410 0x7fbe480c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe48160000 0x7fbe4809eed0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 --2- 172.21.15.146:0/2666423846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe480b7350 0x7fbe4815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 >> 172.21.15.146:0/2666423846 conn(0x7fbe4809b2b0 msgr2=0x7fbe4815f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 shutdown_connections 2024-04-25T15:21:02.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.620+0000 7fbe4ea92640 1 -- 172.21.15.146:0/2666423846 wait complete. 2024-04-25T15:21:02.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-25T15:21:02.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.746+0000 7f24a9da8640 1 Processor -- start 2024-04-25T15:21:02.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.746+0000 7f24a9da8640 1 -- start start 2024-04-25T15:21:02.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a9da8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24a40b9600 0x7f24a415f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a9da8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a406c8c0 con 0x7f24a40b4cc0 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a9da8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a406cba0 con 0x7f24a40d42e0 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a9da8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f24a406d4d0 con 0x7f24a40b9600 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24a40b9600 0x7f24a415f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a2ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f24a40b4cc0 0x7f24a40b8ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41698/0 (socket says 172.21.15.146:41698) 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24a40b9600 0x7f24a415f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48730/0 (socket says 172.21.15.146:48730) 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a2ffd640 1 -- 172.21.15.146:0/1106230132 learned_addr learned my addr 172.21.15.146:0/1106230132 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a3fff640 1 -- 172.21.15.146:0/1106230132 >> v1:172.21.15.177:6789/0 conn(0x7f24a40d42e0 legacy=0x7f24a406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a3fff640 1 -- 172.21.15.146:0/1106230132 >> v1:172.21.15.146:6789/0 conn(0x7f24a40b4cc0 legacy=0x7f24a40b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a27fc640 1 -- 172.21.15.146:0/1106230132 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017711558 0 0) 0x7f24a406c8c0 con 0x7f24a40b4cc0 2024-04-25T15:21:02.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.748+0000 7f24a3fff640 1 -- 172.21.15.146:0/1106230132 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a406ce80 con 0x7f24a40b9600 2024-04-25T15:21:02.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.749+0000 7f24a3fff640 1 --2- 172.21.15.146:0/1106230132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b9600 0x7f24a415f670 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f249c009d90 tx=0x7f249c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea4e0c37ab0da59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:02.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.749+0000 7f24a27fc640 1 -- 172.21.15.146:0/1106230132 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c007740 con 0x7f24a40b9600 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.749+0000 7f24a27fc640 1 -- 172.21.15.146:0/1106230132 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249c0078e0 con 0x7f24a40b9600 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.749+0000 7f24a27fc640 1 -- 172.21.15.146:0/1106230132 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c007cb0 con 0x7f24a40b9600 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.749+0000 7f24a9da8640 1 -- 172.21.15.146:0/1106230132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b9600 msgr2=0x7f24a415f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.749+0000 7f24a9da8640 1 --2- 172.21.15.146:0/1106230132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b9600 0x7f24a415f670 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f249c009d90 tx=0x7f249c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.750+0000 7f24a9da8640 1 -- 172.21.15.146:0/1106230132 shutdown_connections 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.750+0000 7f24a9da8640 1 --2- 172.21.15.146:0/1106230132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b9600 0x7f24a415f670 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.750+0000 7f24a9da8640 1 -- 172.21.15.146:0/1106230132 >> 172.21.15.146:0/1106230132 conn(0x7f24a40d62b0 msgr2=0x7f24a40b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.750+0000 7f24a9da8640 1 -- 172.21.15.146:0/1106230132 shutdown_connections 2024-04-25T15:21:02.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.750+0000 7f24a9da8640 1 -- 172.21.15.146:0/1106230132 wait complete. 2024-04-25T15:21:02.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.751+0000 7f24a9da8640 1 Processor -- start 2024-04-25T15:21:02.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.752+0000 7f24a9da8640 1 -- start start 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a9da8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 0x7f24a41e9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a9da8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24a40b9600 0x7f24a4157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a9da8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24a40d42e0 0x7f24a41f9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a9da8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f24a406c550 con 0x7f24a40b4cc0 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a9da8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24a40b50d0 con 0x7f24a40b9600 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a9da8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f24a40b9a10 con 0x7f24a40d42e0 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 0x7f24a41e9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 0x7f24a41e9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48742/0 (socket says 172.21.15.146:48742) 2024-04-25T15:21:02.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 -- 172.21.15.146:0/3197270191 learned_addr learned my addr 172.21.15.146:0/3197270191 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a2ffd640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24a40b9600 0x7f24a4157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a3fff640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24a40d42e0 0x7f24a41f9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 -- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24a40d42e0 msgr2=0x7f24a41f9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24a40d42e0 0x7f24a41f9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 -- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24a40b9600 msgr2=0x7f24a4157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24a40b9600 0x7f24a4157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 -- 172.21.15.146:0/3197270191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a415ddb0 con 0x7f24a40b4cc0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a3fff640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24a40d42e0 0x7f24a41f9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a2ffd640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24a40b9600 0x7f24a4157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.753+0000 7f24a37fe640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 0x7f24a41e9920 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f249400b420 tx=0x7f249400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.754+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494011830 con 0x7f24a40b4cc0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.754+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2494016070 con 0x7f24a40b4cc0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.754+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a415e090 con 0x7f24a40b4cc0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.754+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494011c10 con 0x7f24a40b4cc0 2024-04-25T15:21:02.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.754+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a415e650 con 0x7f24a40b4cc0 2024-04-25T15:21:02.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.755+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a406a0e0 con 0x7f24a40b4cc0 2024-04-25T15:21:02.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.755+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f249401b460 con 0x7f24a40b4cc0 2024-04-25T15:21:02.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.756+0000 7f248bfff640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24900744b0 0x7f24900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:02.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.756+0000 7f24a2ffd640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24900744b0 0x7f24900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:02.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.756+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f2494094ad0 con 0x7f24a40b4cc0 2024-04-25T15:21:02.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.756+0000 7f24a2ffd640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24900744b0 0x7f24900769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f24a40cb320 tx=0x7f2498007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:02.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:02.760+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2494020020 con 0x7f24a40b4cc0 2024-04-25T15:21:03.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.003+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f24a40d3e00 con 0x7f24a40b4cc0 2024-04-25T15:21:03.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.005+0000 7f248bfff640 1 -- 172.21.15.146:0/3197270191 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f249405f4b0 con 0x7f24a40b4cc0 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24900744b0 msgr2=0x7f24900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24900744b0 0x7f24900769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f24a40cb320 tx=0x7f2498007cc0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 msgr2=0x7f24a41e9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 0x7f24a41e9920 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f249400b420 tx=0x7f249400b8e0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 shutdown_connections 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24900744b0 0x7f24900769c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24a40d42e0 0x7f24a41f9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24a40b9600 0x7f24a4157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 --2- 172.21.15.146:0/3197270191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a40b4cc0 0x7f24a41e9920 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 >> 172.21.15.146:0/3197270191 conn(0x7f24a40d62b0 msgr2=0x7f24a40d4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:03.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 shutdown_connections 2024-04-25T15:21:03.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.008+0000 7f24a9da8640 1 -- 172.21.15.146:0/3197270191 wait complete. 2024-04-25T15:21:03.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-25T15:21:03.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.129+0000 7f2e35114640 1 Processor -- start 2024-04-25T15:21:03.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.129+0000 7f2e35114640 1 -- start start 2024-04-25T15:21:03.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e35114640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 0x7f2e300d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e35114640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2e30161b30 0x7f2e30161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e35114640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e300d1420 con 0x7f2e300cf1d0 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e35114640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2e300ced40 con 0x7f2e30161b30 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e35114640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2e300d1700 con 0x7f2e300d1cc0 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2ed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2e300cf1d0 0x7f2e300b7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41706/0 (socket says 172.21.15.146:41706) 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2ed9d640 1 -- 172.21.15.146:0/2592611356 learned_addr learned my addr 172.21.15.146:0/2592611356 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322644523 0 0) 0x7f2e300d1420 con 0x7f2e300cf1d0 2024-04-25T15:21:03.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2f59e640 1 --2- 172.21.15.146:0/2592611356 >> v2:172.21.15.177:3300/0 conn(0x7f2e30161b30 0x7f2e30161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2e59c640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 0x7f2e300d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e18003520 con 0x7f2e300cf1d0 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 419772618 0 0) 0x7f2e18003520 con 0x7f2e300cf1d0 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 msgr2=0x7f2e300d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 0x7f2e300d20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 >> v2:172.21.15.177:3300/0 conn(0x7f2e30161b30 msgr2=0x7f2e30161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 --2- 172.21.15.146:0/2592611356 >> v2:172.21.15.177:3300/0 conn(0x7f2e30161b30 0x7f2e30161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e300d19e0 con 0x7f2e300cf1d0 2024-04-25T15:21:03.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f2e28001670 con 0x7f2e300cf1d0 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 >> v1:172.21.15.146:6789/0 conn(0x7f2e300cf1d0 legacy=0x7f2e300b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 0x7f2e180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18003f30 0x7f2e18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.131+0000 7f2e2dd9b640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e18005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300ced40 con 0x7f2e18004d40 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300d1700 con 0x7f2e18003f30 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1800ba10 con 0x7f2e300d1cc0 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2f59e640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e18005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2ed9d640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 0x7f2e180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2e59c640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18003f30 0x7f2e18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2f59e640 1 -- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 msgr2=0x7f2e180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2f59e640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e300d1cc0 0x7f2e180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2f59e640 1 -- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18003f30 msgr2=0x7f2e18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2f59e640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18003f30 0x7f2e18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.132+0000 7f2e2f59e640 1 -- 172.21.15.146:0/2592611356 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e18003520 con 0x7f2e18004d40 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e2f59e640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e18005150 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2e3015f6f0 tx=0x7f2e18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218d7f799b22ab56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e20036070 con 0x7f2e18004d40 2024-04-25T15:21:03.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e20031400 con 0x7f2e18004d40 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e2dd9b640 1 -- 172.21.15.146:0/2592611356 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e20009690 con 0x7f2e18004d40 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e35114640 1 -- 172.21.15.146:0/2592611356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 msgr2=0x7f2e18005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e35114640 1 --2- 172.21.15.146:0/2592611356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e18005150 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2e3015f6f0 tx=0x7f2e18005830 comp rx=0 tx=0).stop 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e2f59e640 1 -- 172.21.15.146:0/2592611356 reap_dead start 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e35114640 1 -- 172.21.15.146:0/2592611356 shutdown_connections 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e35114640 1 -- 172.21.15.146:0/2592611356 >> 172.21.15.146:0/2592611356 conn(0x7f2e3009b2b0 msgr2=0x7f2e300b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e35114640 1 -- 172.21.15.146:0/2592611356 shutdown_connections 2024-04-25T15:21:03.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.133+0000 7f2e35114640 1 -- 172.21.15.146:0/2592611356 wait complete. 2024-04-25T15:21:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.136+0000 7f2e35114640 1 Processor -- start 2024-04-25T15:21:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.136+0000 7f2e35114640 1 -- start start 2024-04-25T15:21:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e35114640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e30160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e35114640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3015e610 0x7f2e300c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e35114640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e3015ea20 0x7f2e300c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e35114640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3015e310 con 0x7f2e18004d40 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e35114640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300cf5e0 con 0x7f2e3015e610 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e35114640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3015e490 con 0x7f2e3015ea20 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e30160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e30160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48764/0 (socket says 172.21.15.146:48764) 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 -- 172.21.15.146:0/2341116478 learned_addr learned my addr 172.21.15.146:0/2341116478 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2e59c640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3015e610 0x7f2e300c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2f59e640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e3015ea20 0x7f2e300c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 -- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e3015ea20 msgr2=0x7f2e300c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e3015ea20 0x7f2e300c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 -- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3015e610 msgr2=0x7f2e300c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3015e610 0x7f2e300c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 -- 172.21.15.146:0/2341116478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3015f220 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.137+0000 7f2e2ed9d640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e30160890 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007340 tx=0x7f2e28007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.138+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800fa00 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.138+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e28018070 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.138+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3015f500 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.138+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800fde0 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.138+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3015fac0 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.139+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e3006a0e0 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.139+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2e2801d460 con 0x7f2e18004d40 2024-04-25T15:21:03.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.140+0000 7f2e177fe640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2e1c0744e0 0x7f2e1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.140+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f2e280951c0 con 0x7f2e18004d40 2024-04-25T15:21:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.140+0000 7f2e2e59c640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2e1c0744e0 0x7f2e1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.140+0000 7f2e2e59c640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2e1c0744e0 0x7f2e1c0769f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2e300c8850 tx=0x7f2e2400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.145+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e2805fba0 con 0x7f2e18004d40 2024-04-25T15:21:03.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.390+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f2e300d2a70 con 0x7f2e18004d40 2024-04-25T15:21:03.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.393+0000 7f2e177fe640 1 -- 172.21.15.146:0/2341116478 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f2e2805f4a0 con 0x7f2e18004d40 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2e1c0744e0 msgr2=0x7f2e1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2e1c0744e0 0x7f2e1c0769f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2e300c8850 tx=0x7f2e2400b040 comp rx=0 tx=0).stop 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 msgr2=0x7f2e30160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e30160890 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007340 tx=0x7f2e28007800 comp rx=0 tx=0).stop 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 shutdown_connections 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2e1c0744e0 0x7f2e1c0769f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e3015ea20 0x7f2e300c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3015e610 0x7f2e300c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 --2- 172.21.15.146:0/2341116478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18004d40 0x7f2e30160890 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 >> 172.21.15.146:0/2341116478 conn(0x7f2e3009b2b0 msgr2=0x7f2e3006d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 shutdown_connections 2024-04-25T15:21:03.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.395+0000 7f2e35114640 1 -- 172.21.15.146:0/2341116478 wait complete. 2024-04-25T15:21:03.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-25T15:21:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.515+0000 7ff4fa11f640 1 Processor -- start 2024-04-25T15:21:03.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.515+0000 7ff4fa11f640 1 -- start start 2024-04-25T15:21:03.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.516+0000 7ff4fa11f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff4f40d21f0 0x7ff4f415ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.516+0000 7ff4fa11f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff4f406f2c0 0x7ff4f40b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.516+0000 7ff4fa11f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.516+0000 7ff4fa11f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f40d1d60 con 0x7ff4f406f2c0 2024-04-25T15:21:03.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.516+0000 7ff4fa11f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f409c2a0 con 0x7ff4f40d21f0 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.516+0000 7ff4fa11f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f409bce0 con 0x7ff4f406f6d0 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff4f406f2c0 0x7ff4f40b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff4f406f2c0 0x7ff4f40b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48772/0 (socket says 172.21.15.146:48772) 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 -- 172.21.15.146:0/1915118760 learned_addr learned my addr 172.21.15.146:0/1915118760 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f3fff640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f37fe640 1 --2- 172.21.15.146:0/1915118760 >> v2:172.21.15.177:3300/0 conn(0x7ff4f40d21f0 0x7ff4f415ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 -- 172.21.15.146:0/1915118760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 msgr2=0x7ff4f40b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 -- 172.21.15.146:0/1915118760 >> v2:172.21.15.177:3300/0 conn(0x7ff4f40d21f0 msgr2=0x7ff4f415ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 --2- 172.21.15.146:0/1915118760 >> v2:172.21.15.177:3300/0 conn(0x7ff4f40d21f0 0x7ff4f415ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f2ffd640 1 -- 172.21.15.146:0/1915118760 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f409bfc0 con 0x7ff4f406f2c0 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f37fe640 1 --2- 172.21.15.146:0/1915118760 >> v2:172.21.15.177:3300/0 conn(0x7ff4f40d21f0 0x7ff4f415ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.517+0000 7ff4f3fff640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:03.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4f2ffd640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f406f2c0 0x7ff4f40b36d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40d26c0 tx=0x7ff4e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea51586cdbc718ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4f27fc640 1 -- 172.21.15.146:0/1915118760 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8037070 con 0x7ff4f406f2c0 2024-04-25T15:21:03.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4f27fc640 1 -- 172.21.15.146:0/1915118760 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e8032440 con 0x7ff4f406f2c0 2024-04-25T15:21:03.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4f27fc640 1 -- 172.21.15.146:0/1915118760 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8032810 con 0x7ff4f406f2c0 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 -- 172.21.15.146:0/1915118760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f406f2c0 msgr2=0x7ff4f40b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f406f2c0 0x7ff4f40b36d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40d26c0 tx=0x7ff4e802a270 comp rx=0 tx=0).stop 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 -- 172.21.15.146:0/1915118760 shutdown_connections 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/1915118760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f406f2c0 0x7ff4f40b36d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/1915118760 >> v2:172.21.15.177:3300/0 conn(0x7ff4f40d21f0 0x7ff4f415ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.518+0000 7ff4fa11f640 1 -- 172.21.15.146:0/1915118760 >> 172.21.15.146:0/1915118760 conn(0x7ff4f40b7530 msgr2=0x7ff4f40b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.519+0000 7ff4fa11f640 1 -- 172.21.15.146:0/1915118760 shutdown_connections 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.519+0000 7ff4fa11f640 1 -- 172.21.15.146:0/1915118760 wait complete. 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.520+0000 7ff4fa11f640 1 Processor -- start 2024-04-25T15:21:03.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.520+0000 7ff4fa11f640 1 -- start start 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4fa11f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4f406f2c0 0x7ff4f41f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4fa11f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4fa11f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 0x7ff4f41f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4fa11f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f409b580 con 0x7ff4f40d21f0 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4fa11f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f409b400 con 0x7ff4f406f2c0 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4fa11f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f40b21b0 con 0x7ff4f406f6d0 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 0x7ff4f41f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 0x7ff4f41f6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48776/0 (socket says 172.21.15.146:48776) 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 -- 172.21.15.146:0/3433646828 learned_addr learned my addr 172.21.15.146:0/3433646828 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f2ffd640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f37fe640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4f406f2c0 0x7ff4f41f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 -- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 msgr2=0x7ff4f40cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 -- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4f406f2c0 msgr2=0x7ff4f41f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4f406f2c0 0x7ff4f41f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 -- 172.21.15.146:0/3433646828 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f41f53a0 con 0x7ff4f40d21f0 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4f3fff640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 0x7ff4f41f6c40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec007df0 tx=0x7ff4ec00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.522+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec01a070 con 0x7ff4f40d21f0 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.523+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ec003990 con 0x7ff4f40d21f0 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.523+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f41582b0 con 0x7ff4f40d21f0 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.523+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec003d80 con 0x7ff4f40d21f0 2024-04-25T15:21:03.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.523+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f4158870 con 0x7ff4f40d21f0 2024-04-25T15:21:03.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.523+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f406a0e0 con 0x7ff4f40d21f0 2024-04-25T15:21:03.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.524+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff4ec01d070 con 0x7ff4f40d21f0 2024-04-25T15:21:03.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.524+0000 7ff4dbfff640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4e00744b0 0x7ff4e00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.525+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7ff4ec0953a0 con 0x7ff4f40d21f0 2024-04-25T15:21:03.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.525+0000 7ff4f37fe640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4e00744b0 0x7ff4e00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.525+0000 7ff4f37fe640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4e00744b0 0x7ff4e00769c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.529+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4ec010070 con 0x7ff4f40d21f0 2024-04-25T15:21:03.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.768+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7ff4f40cc840 con 0x7ff4f40d21f0 2024-04-25T15:21:03.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.770+0000 7ff4dbfff640 1 -- 172.21.15.146:0/3433646828 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7ff4ec05fc70 con 0x7ff4f40d21f0 2024-04-25T15:21:03.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4e00744b0 msgr2=0x7ff4e00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4e00744b0 0x7ff4e00769c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).stop 2024-04-25T15:21:03.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 msgr2=0x7ff4f41f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 0x7ff4f41f6c40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec007df0 tx=0x7ff4ec00d4a0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 shutdown_connections 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4e00744b0 0x7ff4e00769c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4f40d21f0 0x7ff4f41f6c40 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4f406f6d0 0x7ff4f40cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 --2- 172.21.15.146:0/3433646828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4f406f2c0 0x7ff4f41f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 >> 172.21.15.146:0/3433646828 conn(0x7ff4f40b7530 msgr2=0x7ff4f40af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 shutdown_connections 2024-04-25T15:21:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.773+0000 7ff4fa11f640 1 -- 172.21.15.146:0/3433646828 wait complete. 2024-04-25T15:21:03.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-25T15:21:03.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.893+0000 7f9542e92640 1 Processor -- start 2024-04-25T15:21:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.893+0000 7f9542e92640 1 -- start start 2024-04-25T15:21:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f9542e92640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f953c0cfbe0 0x7f953c0cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f9542e92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f953c0d19d0 0x7f953c0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f9542e92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c0b4a60 con 0x7f953c0b3920 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f9542e92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f953c0b5020 con 0x7f953c0d19d0 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f9542e92640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f953c0b4d40 con 0x7f953c0cfbe0 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f953bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f953c0d19d0 0x7f953c0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f954142f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f953c0b3920 0x7f953c0b3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41712/0 (socket says 172.21.15.146:41712) 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.894+0000 7f954142f640 1 -- 172.21.15.146:0/1035462366 learned_addr learned my addr 172.21.15.146:0/1035462366 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f9540c2e640 1 --2- 172.21.15.146:0/1035462366 >> v2:172.21.15.177:3300/0 conn(0x7f953c0cfbe0 0x7f953c0cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953bfff640 1 -- 172.21.15.146:0/1035462366 >> v2:172.21.15.177:3300/0 conn(0x7f953c0cfbe0 msgr2=0x7f953c0cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953bfff640 1 --2- 172.21.15.146:0/1035462366 >> v2:172.21.15.177:3300/0 conn(0x7f953c0cfbe0 0x7f953c0cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953bfff640 1 -- 172.21.15.146:0/1035462366 >> v1:172.21.15.146:6789/0 conn(0x7f953c0b3920 legacy=0x7f953c0b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953bfff640 1 -- 172.21.15.146:0/1035462366 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953c0cead0 con 0x7f953c0d19d0 2024-04-25T15:21:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953b7fe640 1 -- 172.21.15.146:0/1035462366 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415006287 0 0) 0x7f953c0b4a60 con 0x7f953c0b3920 2024-04-25T15:21:03.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953bfff640 1 --2- 172.21.15.146:0/1035462366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c0d1de0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f953c075260 tx=0x7f952802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39e3ee785f1fe90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953b7fe640 1 -- 172.21.15.146:0/1035462366 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528037070 con 0x7f953c0d19d0 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953b7fe640 1 -- 172.21.15.146:0/1035462366 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9528032440 con 0x7f953c0d19d0 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.895+0000 7f953b7fe640 1 -- 172.21.15.146:0/1035462366 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528032810 con 0x7f953c0d19d0 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 -- 172.21.15.146:0/1035462366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 msgr2=0x7f953c0d1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 --2- 172.21.15.146:0/1035462366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c0d1de0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f953c075260 tx=0x7f952802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 -- 172.21.15.146:0/1035462366 shutdown_connections 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 --2- 172.21.15.146:0/1035462366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c0d1de0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 --2- 172.21.15.146:0/1035462366 >> v2:172.21.15.177:3300/0 conn(0x7f953c0cfbe0 0x7f953c0cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 -- 172.21.15.146:0/1035462366 >> 172.21.15.146:0/1035462366 conn(0x7f953c0b73b0 msgr2=0x7f953c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 -- 172.21.15.146:0/1035462366 shutdown_connections 2024-04-25T15:21:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.896+0000 7f9542e92640 1 -- 172.21.15.146:0/1035462366 wait complete. 2024-04-25T15:21:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.898+0000 7f9542e92640 1 Processor -- start 2024-04-25T15:21:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.898+0000 7f9542e92640 1 -- start start 2024-04-25T15:21:03.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.899+0000 7f9542e92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f953c0b3920 0x7f953c0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.899+0000 7f9542e92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f953c0cfbe0 0x7f953c1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.899+0000 7f9542e92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c06de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.899+0000 7f9542e92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f953c0cf7a0 con 0x7f953c0d19d0 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.899+0000 7f9542e92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f953c0d2320 con 0x7f953c0cfbe0 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.899+0000 7f9542e92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f953c153090 con 0x7f953c0b3920 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c06de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c06de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48792/0 (socket says 172.21.15.146:48792) 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 -- 172.21.15.146:0/3392566206 learned_addr learned my addr 172.21.15.146:0/3392566206 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f9540c2e640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f953c0b3920 0x7f953c0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f953bfff640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f953c0cfbe0 0x7f953c1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 -- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f953c0b3920 msgr2=0x7f953c0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f953c0b3920 0x7f953c0c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 -- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f953c0cfbe0 msgr2=0x7f953c1538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f953c0cfbe0 0x7f953c1538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 -- 172.21.15.146:0/3392566206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953c151590 con 0x7f953c0d19d0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f954142f640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c06de00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f953400c730 tx=0x7f953400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9534018070 con 0x7f953c0d19d0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f953400ce40 con 0x7f953c0d19d0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f953c0c6390 con 0x7f953c0d19d0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400a880 con 0x7f953c0d19d0 2024-04-25T15:21:03.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.900+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f953c152dd0 con 0x7f953c0d19d0 2024-04-25T15:21:03.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.901+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f953c06a0e0 con 0x7f953c0d19d0 2024-04-25T15:21:03.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.902+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f953401b070 con 0x7f953c0d19d0 2024-04-25T15:21:03.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.902+0000 7f95397fa640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f95200744e0 0x7f95200769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:03.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.902+0000 7f9540c2e640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f95200744e0 0x7f95200769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:03.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.902+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f9534094510 con 0x7f953c0d19d0 2024-04-25T15:21:03.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.903+0000 7f9540c2e640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f95200744e0 0x7f95200769f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9530005f80 tx=0x7f9530008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:03.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:03.907+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f953405e810 con 0x7f953c0d19d0 2024-04-25T15:21:04.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.153+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f953c0c6d20 con 0x7f953c0d19d0 2024-04-25T15:21:04.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.156+0000 7f95397fa640 1 -- 172.21.15.146:0/3392566206 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f9534062b10 con 0x7f953c0d19d0 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f95200744e0 msgr2=0x7f95200769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f95200744e0 0x7f95200769f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9530005f80 tx=0x7f9530008040 comp rx=0 tx=0).stop 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 msgr2=0x7f953c06de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c06de00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f953400c730 tx=0x7f953400cbf0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 shutdown_connections 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f95200744e0 0x7f95200769f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f953c0d19d0 0x7f953c06de00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f953c0cfbe0 0x7f953c1538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 --2- 172.21.15.146:0/3392566206 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f953c0b3920 0x7f953c0c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 >> 172.21.15.146:0/3392566206 conn(0x7f953c0b73b0 msgr2=0x7f953c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:04.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 shutdown_connections 2024-04-25T15:21:04.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.158+0000 7f9542e92640 1 -- 172.21.15.146:0/3392566206 wait complete. 2024-04-25T15:21:04.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-25T15:21:04.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.279+0000 7fc1626af640 1 Processor -- start 2024-04-25T15:21:04.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.279+0000 7fc1626af640 1 -- start start 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.280+0000 7fc1626af640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc15c0b5950 0x7fc15c0d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc1626af640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc15c15ffa0 0x7fc15c0b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc1626af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc15c0d2230 con 0x7fc15c0b6610 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc1626af640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc15c0b2f50 con 0x7fc15c15ffa0 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc1626af640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc15c0b2ac0 con 0x7fc15c0b5950 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc160c4c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc15c15ffa0 0x7fc15c0b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc15bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc15c0b6610 0x7fc15c0b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41714/0 (socket says 172.21.15.146:41714) 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc15bfff640 1 -- 172.21.15.146:0/3300288527 learned_addr learned my addr 172.21.15.146:0/3300288527 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:04.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc15b7fe640 1 --2- 172.21.15.146:0/3300288527 >> v2:172.21.15.177:3300/0 conn(0x7fc15c0b5950 0x7fc15c0d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc160c4c640 1 -- 172.21.15.146:0/3300288527 >> v2:172.21.15.177:3300/0 conn(0x7fc15c0b5950 msgr2=0x7fc15c0d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:04.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc160c4c640 1 --2- 172.21.15.146:0/3300288527 >> v2:172.21.15.177:3300/0 conn(0x7fc15c0b5950 0x7fc15c0d2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc160c4c640 1 -- 172.21.15.146:0/3300288527 >> v1:172.21.15.146:6789/0 conn(0x7fc15c0b6610 legacy=0x7fc15c0b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc160c4c640 1 -- 172.21.15.146:0/3300288527 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c0cfa80 con 0x7fc15c15ffa0 2024-04-25T15:21:04.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.281+0000 7fc15affd640 1 -- 172.21.15.146:0/3300288527 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331003289 0 0) 0x7fc15c0d2230 con 0x7fc15c0b6610 2024-04-25T15:21:04.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.282+0000 7fc160c4c640 1 --2- 172.21.15.146:0/3300288527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c0b7cc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c7080 tx=0x7fc15402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd2a04e26eb44e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:04.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.282+0000 7fc15affd640 1 -- 172.21.15.146:0/3300288527 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154007740 con 0x7fc15c15ffa0 2024-04-25T15:21:04.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.282+0000 7fc15affd640 1 -- 172.21.15.146:0/3300288527 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1540078e0 con 0x7fc15c15ffa0 2024-04-25T15:21:04.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.282+0000 7fc15affd640 1 -- 172.21.15.146:0/3300288527 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154007cb0 con 0x7fc15c15ffa0 2024-04-25T15:21:04.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 -- 172.21.15.146:0/3300288527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 msgr2=0x7fc15c0b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 --2- 172.21.15.146:0/3300288527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c0b7cc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c7080 tx=0x7fc15402a680 comp rx=0 tx=0).stop 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 -- 172.21.15.146:0/3300288527 shutdown_connections 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 --2- 172.21.15.146:0/3300288527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c0b7cc0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 --2- 172.21.15.146:0/3300288527 >> v2:172.21.15.177:3300/0 conn(0x7fc15c0b5950 0x7fc15c0d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 -- 172.21.15.146:0/3300288527 >> 172.21.15.146:0/3300288527 conn(0x7fc15c0d0240 msgr2=0x7fc15c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 -- 172.21.15.146:0/3300288527 shutdown_connections 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.283+0000 7fc1626af640 1 -- 172.21.15.146:0/3300288527 wait complete. 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.284+0000 7fc1626af640 1 Processor -- start 2024-04-25T15:21:04.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.285+0000 7fc1626af640 1 -- start start 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc1626af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc15c0b6610 0x7fc15c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc1626af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c1621f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc1626af640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc15c1f96f0 0x7fc15c149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc1626af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c0d1da0 con 0x7fc15c15ffa0 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc1626af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c1603b0 con 0x7fc15c0b6610 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc1626af640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c0d1f20 con 0x7fc15c1f96f0 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c1621f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c1621f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48822/0 (socket says 172.21.15.146:48822) 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 -- 172.21.15.146:0/1579611758 learned_addr learned my addr 172.21.15.146:0/1579611758 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15bfff640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc15c0b6610 0x7fc15c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc160c4c640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc15c1f96f0 0x7fc15c149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 -- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc15c1f96f0 msgr2=0x7fc15c149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc15c1f96f0 0x7fc15c149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 -- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc15c0b6610 msgr2=0x7fc15c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc15c0b6610 0x7fc15c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 -- 172.21.15.146:0/1579611758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c1f9ed0 con 0x7fc15c15ffa0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15bfff640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc15c0b6610 0x7fc15c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.286+0000 7fc15b7fe640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c1621f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e870 tx=0x7fc14400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.287+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144017070 con 0x7fc15c15ffa0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.287+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14400c940 con 0x7fc15c15ffa0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.287+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14400cd30 con 0x7fc15c15ffa0 2024-04-25T15:21:04.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.287+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c1fa1b0 con 0x7fc15c15ffa0 2024-04-25T15:21:04.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.287+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c155940 con 0x7fc15c15ffa0 2024-04-25T15:21:04.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.288+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15c155730 con 0x7fc15c15ffa0 2024-04-25T15:21:04.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.288+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc14401a070 con 0x7fc15c15ffa0 2024-04-25T15:21:04.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.289+0000 7fc158ff9640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc13c0744e0 0x7fc13c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.289+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fc144094f40 con 0x7fc15c15ffa0 2024-04-25T15:21:04.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.289+0000 7fc15bfff640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc13c0744e0 0x7fc13c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.289+0000 7fc15bfff640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc13c0744e0 0x7fc13c0769f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc14c000f60 tx=0x7fc14c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:04.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.293+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc14405f890 con 0x7fc15c15ffa0 2024-04-25T15:21:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.534+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fc15c06a0e0 con 0x7fc15c15ffa0 2024-04-25T15:21:04.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.537+0000 7fc158ff9640 1 -- 172.21.15.146:0/1579611758 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fc14405f190 con 0x7fc15c15ffa0 2024-04-25T15:21:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.539+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc13c0744e0 msgr2=0x7fc13c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.539+0000 7fc1626af640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc13c0744e0 0x7fc13c0769f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc14c000f60 tx=0x7fc14c009040 comp rx=0 tx=0).stop 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.539+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 msgr2=0x7fc15c1621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.539+0000 7fc1626af640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c1621f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e870 tx=0x7fc14400ed30 comp rx=0 tx=0).stop 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 shutdown_connections 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc13c0744e0 0x7fc13c0769f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc15c1f96f0 0x7fc15c149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc15c15ffa0 0x7fc15c1621f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 --2- 172.21.15.146:0/1579611758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc15c0b6610 0x7fc15c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 >> 172.21.15.146:0/1579611758 conn(0x7fc15c0d0240 msgr2=0x7fc15c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 shutdown_connections 2024-04-25T15:21:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.540+0000 7fc1626af640 1 -- 172.21.15.146:0/1579611758 wait complete. 2024-04-25T15:21:04.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-25T15:21:04.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.660+0000 7efde511f640 1 Processor -- start 2024-04-25T15:21:04.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.660+0000 7efde511f640 1 -- start start 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.661+0000 7efde511f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 0x7efde00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.661+0000 7efde511f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efde00c6cc0 0x7efde0164bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.661+0000 7efde511f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efde0160410 con 0x7efde00c68b0 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.661+0000 7efde511f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7efde006c8c0 con 0x7efde00c6cc0 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.661+0000 7efde511f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efde00b2d50 con 0x7efde006da10 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efdded9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 0x7efde00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efdded9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 0x7efde00d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:42138/0 (socket says 172.21.15.146:42138) 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efdded9d640 1 -- 172.21.15.146:0/3485585542 learned_addr learned my addr 172.21.15.146:0/3485585542 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efddf59e640 1 --2- 172.21.15.146:0/3485585542 >> v2:172.21.15.177:3300/0 conn(0x7efde00c6cc0 0x7efde0164bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efddf59e640 1 -- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 msgr2=0x7efde00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:04.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efddf59e640 1 --2- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 0x7efde00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efddf59e640 1 -- 172.21.15.146:0/3485585542 >> v1:172.21.15.177:6789/0 conn(0x7efde00c68b0 legacy=0x7efde006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efddf59e640 1 -- 172.21.15.146:0/3485585542 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde00b3030 con 0x7efde00c6cc0 2024-04-25T15:21:04.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.662+0000 7efdded9d640 1 --2- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 0x7efde00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.663+0000 7efddf59e640 1 --2- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c6cc0 0x7efde0164bc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efde006ca40 tx=0x7efdd802a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3892fc9391bf061 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.663+0000 7efdde59c640 1 -- 172.21.15.146:0/3485585542 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8007740 con 0x7efde00c6cc0 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.663+0000 7efdde59c640 1 -- 172.21.15.146:0/3485585542 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd80078e0 con 0x7efde00c6cc0 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.663+0000 7efdde59c640 1 -- 172.21.15.146:0/3485585542 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8007cb0 con 0x7efde00c6cc0 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 -- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c6cc0 msgr2=0x7efde0164bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 --2- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c6cc0 0x7efde0164bc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efde006ca40 tx=0x7efdd802a5b0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 -- 172.21.15.146:0/3485585542 shutdown_connections 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 --2- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c6cc0 0x7efde0164bc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 --2- 172.21.15.146:0/3485585542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde006da10 0x7efde00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 -- 172.21.15.146:0/3485585542 >> 172.21.15.146:0/3485585542 conn(0x7efde009b2b0 msgr2=0x7efde00b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:04.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 -- 172.21.15.146:0/3485585542 shutdown_connections 2024-04-25T15:21:04.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.664+0000 7efde511f640 1 -- 172.21.15.146:0/3485585542 wait complete. 2024-04-25T15:21:04.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.666+0000 7efde511f640 1 Processor -- start 2024-04-25T15:21:04.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.666+0000 7efde511f640 1 -- start start 2024-04-25T15:21:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efde511f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 0x7efde014b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efde511f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c68b0 0x7efde00af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efde511f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde00c6cc0 0x7efde015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efde511f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efde00d2760 con 0x7efde006da10 2024-04-25T15:21:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efde511f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efde006c550 con 0x7efde00c68b0 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efde511f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efde006d460 con 0x7efde00c6cc0 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 0x7efde014b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 0x7efde014b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48828/0 (socket says 172.21.15.146:48828) 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 -- 172.21.15.146:0/3031332871 learned_addr learned my addr 172.21.15.146:0/3031332871 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efddf59e640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde00c6cc0 0x7efde015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdd7fff640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c68b0 0x7efde00af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 -- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde00c6cc0 msgr2=0x7efde015c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde00c6cc0 0x7efde015c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 -- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c68b0 msgr2=0x7efde00af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.667+0000 7efdded9d640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c68b0 0x7efde00af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efdded9d640 1 -- 172.21.15.146:0/3031332871 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde01f6250 con 0x7efde006da10 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efdded9d640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 0x7efde014b6e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efdd000e930 tx=0x7efdd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0017070 con 0x7efde006da10 2024-04-25T15:21:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd000ca00 con 0x7efde006da10 2024-04-25T15:21:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde014c0f0 con 0x7efde006da10 2024-04-25T15:21:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd000cdf0 con 0x7efde006da10 2024-04-25T15:21:04.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.668+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde00b0180 con 0x7efde006da10 2024-04-25T15:21:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.669+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde006a0e0 con 0x7efde006da10 2024-04-25T15:21:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.669+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efdd001a070 con 0x7efde006da10 2024-04-25T15:21:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.670+0000 7efdd77fe640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efdc00744b0 0x7efdc00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.670+0000 7efdd7fff640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efdc00744b0 0x7efdc00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.670+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7efdd0095010 con 0x7efde006da10 2024-04-25T15:21:04.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.670+0000 7efdd7fff640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efdc00744b0 0x7efdc00769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdc8007fd0 tx=0x7efdc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.674+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efde01f6250 con 0x7efde006da10 2024-04-25T15:21:04.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.920+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7efde00b7130 con 0x7efde006da10 2024-04-25T15:21:04.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.922+0000 7efdd77fe640 1 -- 172.21.15.146:0/3031332871 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7efdd005f8e0 con 0x7efde006da10 2024-04-25T15:21:04.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efdc00744b0 msgr2=0x7efdc00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efdc00744b0 0x7efdc00769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdc8007fd0 tx=0x7efdc8009040 comp rx=0 tx=0).stop 2024-04-25T15:21:04.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 msgr2=0x7efde014b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:04.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 0x7efde014b6e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efdd000e930 tx=0x7efdd000edf0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 shutdown_connections 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efdc00744b0 0x7efdc00769c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efde00c6cc0 0x7efde015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efde00c68b0 0x7efde00af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 --2- 172.21.15.146:0/3031332871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efde006da10 0x7efde014b6e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 >> 172.21.15.146:0/3031332871 conn(0x7efde009b2b0 msgr2=0x7efde00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 shutdown_connections 2024-04-25T15:21:04.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:04.924+0000 7efde511f640 1 -- 172.21.15.146:0/3031332871 wait complete. 2024-04-25T15:21:04.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:05.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.044+0000 7ff48c80c640 1 Processor -- start 2024-04-25T15:21:05.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.044+0000 7ff48c80c640 1 -- start start 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48c80c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff48806ee60 0x7ff4880b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48c80c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff488163eb0 0x7ff4881642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48c80c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4880b8eb0 con 0x7ff488164800 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48c80c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff48808d730 con 0x7ff48806ee60 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48c80c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff48806f440 con 0x7ff488163eb0 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff48806ee60 0x7ff4880b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff48806ee60 0x7ff4880b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48832/0 (socket says 172.21.15.146:48832) 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 -- 172.21.15.146:0/4220646580 learned_addr learned my addr 172.21.15.146:0/4220646580 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 -- 172.21.15.146:0/4220646580 >> v2:172.21.15.177:3300/0 conn(0x7ff488163eb0 msgr2=0x7ff4881642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:05.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff485d9b640 1 --2- 172.21.15.146:0/4220646580 >> v2:172.21.15.177:3300/0 conn(0x7ff488163eb0 0x7ff4881642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 --2- 172.21.15.146:0/4220646580 >> v2:172.21.15.177:3300/0 conn(0x7ff488163eb0 0x7ff4881642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 -- 172.21.15.146:0/4220646580 >> v1:172.21.15.177:6789/0 conn(0x7ff488164800 legacy=0x7ff488160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff48659c640 1 -- 172.21.15.146:0/4220646580 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48808ce00 con 0x7ff48806ee60 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.046+0000 7ff485d9b640 1 --2- 172.21.15.146:0/4220646580 >> v2:172.21.15.177:3300/0 conn(0x7ff488163eb0 0x7ff4881642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.047+0000 7ff48659c640 1 --2- 172.21.15.146:0/4220646580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48806ee60 0x7ff4880b2ba0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff4880b30e0 tx=0x7ff47802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbbe66eff3f3f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.047+0000 7ff48559a640 1 -- 172.21.15.146:0/4220646580 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478004030 con 0x7ff48806ee60 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.047+0000 7ff48559a640 1 -- 172.21.15.146:0/4220646580 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4780041d0 con 0x7ff48806ee60 2024-04-25T15:21:05.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.047+0000 7ff48559a640 1 -- 172.21.15.146:0/4220646580 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478003610 con 0x7ff48806ee60 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 -- 172.21.15.146:0/4220646580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48806ee60 msgr2=0x7ff4880b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 --2- 172.21.15.146:0/4220646580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48806ee60 0x7ff4880b2ba0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff4880b30e0 tx=0x7ff47802a210 comp rx=0 tx=0).stop 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 -- 172.21.15.146:0/4220646580 shutdown_connections 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 --2- 172.21.15.146:0/4220646580 >> v2:172.21.15.177:3300/0 conn(0x7ff488163eb0 0x7ff4881642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 --2- 172.21.15.146:0/4220646580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48806ee60 0x7ff4880b2ba0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 -- 172.21.15.146:0/4220646580 >> 172.21.15.146:0/4220646580 conn(0x7ff4880b52e0 msgr2=0x7ff4880b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 -- 172.21.15.146:0/4220646580 shutdown_connections 2024-04-25T15:21:05.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.048+0000 7ff48c80c640 1 -- 172.21.15.146:0/4220646580 wait complete. 2024-04-25T15:21:05.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.050+0000 7ff48c80c640 1 Processor -- start 2024-04-25T15:21:05.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.050+0000 7ff48c80c640 1 -- start start 2024-04-25T15:21:05.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48c80c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff48806ee60 0x7ff48815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48c80c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488163eb0 0x7ff4881ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48c80c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 0x7ff4881ea950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48c80c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff48808dbc0 con 0x7ff488164800 2024-04-25T15:21:05.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48c80c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff48808d3c0 con 0x7ff488163eb0 2024-04-25T15:21:05.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48c80c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4880d2b70 con 0x7ff48806ee60 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff486d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 0x7ff4881ea950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff486d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 0x7ff4881ea950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48842/0 (socket says 172.21.15.146:48842) 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff486d9d640 1 -- 172.21.15.146:0/1996824357 learned_addr learned my addr 172.21.15.146:0/1996824357 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff485d9b640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488163eb0 0x7ff4881ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.051+0000 7ff48659c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff48806ee60 0x7ff48815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff486d9d640 1 -- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff48806ee60 msgr2=0x7ff48815d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff486d9d640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff48806ee60 0x7ff48815d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff486d9d640 1 -- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488163eb0 msgr2=0x7ff4881ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff486d9d640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488163eb0 0x7ff4881ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff486d9d640 1 -- 172.21.15.146:0/1996824357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff488161db0 con 0x7ff488164800 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff486d9d640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 0x7ff4881ea950 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff48000ba40 tx=0x7ff48000bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48000cb80 con 0x7ff488164800 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48000cd20 con 0x7ff488164800 2024-04-25T15:21:05.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff488162090 con 0x7ff488164800 2024-04-25T15:21:05.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480003bb0 con 0x7ff488164800 2024-04-25T15:21:05.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.052+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff488162650 con 0x7ff488164800 2024-04-25T15:21:05.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.053+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48806a0e0 con 0x7ff488164800 2024-04-25T15:21:05.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.054+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff480014440 con 0x7ff488164800 2024-04-25T15:21:05.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.054+0000 7ff476ffd640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4680744e0 0x7ff4680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.054+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7ff480094de0 con 0x7ff488164800 2024-04-25T15:21:05.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.054+0000 7ff48659c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4680744e0 0x7ff4680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.054+0000 7ff48659c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4680744e0 0x7ff4680769f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff478033000 tx=0x7ff47802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.059+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff48005f0e0 con 0x7ff488164800 2024-04-25T15:21:05.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.238+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4881ea1f0 con 0x7ff488164800 2024-04-25T15:21:05.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.239+0000 7ff476ffd640 1 -- 172.21.15.146:0/1996824357 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13212 (secure 0 0 0) 0x7ff4800633e0 con 0x7ff488164800 2024-04-25T15:21:05.239 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:05.240 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":35,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:02.243391+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4680744e0 msgr2=0x7ff4680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4680744e0 0x7ff4680769f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff478033000 tx=0x7ff47802e000 comp rx=0 tx=0).stop 2024-04-25T15:21:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 msgr2=0x7ff4881ea950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 0x7ff4881ea950 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff48000ba40 tx=0x7ff48000bf00 comp rx=0 tx=0).stop 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 shutdown_connections 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4680744e0 0x7ff4680769f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff488164800 0x7ff4881ea950 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488163eb0 0x7ff4881ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 --2- 172.21.15.146:0/1996824357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff48806ee60 0x7ff48815d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 >> 172.21.15.146:0/1996824357 conn(0x7ff4880b52e0 msgr2=0x7ff4880cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 shutdown_connections 2024-04-25T15:21:05.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.241+0000 7ff48c80c640 1 -- 172.21.15.146:0/1996824357 wait complete. 2024-04-25T15:21:05.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:05.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:05.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.365+0000 7fb275049640 1 Processor -- start 2024-04-25T15:21:05.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.365+0000 7fb275049640 1 -- start start 2024-04-25T15:21:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb275049640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 0x7fb270073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb275049640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 0x7fb270074020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb275049640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb27015d940 con 0x7fb2700ae660 2024-04-25T15:21:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb275049640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb27006fe30 con 0x7fb2700b6460 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb275049640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb2700702c0 con 0x7fb27015d480 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb26f59e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 0x7fb270074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 0x7fb270073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb26f59e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 0x7fb270074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:37702/0 (socket says 172.21.15.146:37702) 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 0x7fb270073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:42154/0 (socket says 172.21.15.146:42154) 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.367+0000 7fb26f59e640 1 -- 172.21.15.146:0/407432096 learned_addr learned my addr 172.21.15.146:0/407432096 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.368+0000 7fb26ed9d640 1 -- 172.21.15.146:0/407432096 >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 msgr2=0x7fb270074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.368+0000 7fb26ed9d640 1 --2- 172.21.15.146:0/407432096 >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 0x7fb270074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.368+0000 7fb26ed9d640 1 -- 172.21.15.146:0/407432096 >> v1:172.21.15.177:6789/0 conn(0x7fb2700ae660 legacy=0x7fb27015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.368+0000 7fb26ed9d640 1 -- 172.21.15.146:0/407432096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27006f500 con 0x7fb27015d480 2024-04-25T15:21:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.368+0000 7fb26f59e640 1 --2- 172.21.15.146:0/407432096 >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 0x7fb270074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.368+0000 7fb26ed9d640 1 --2- 172.21.15.146:0/407432096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 0x7fb270073ae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb26000aea0 tx=0x7fb26002b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef13e8e7cbf3d26f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb26dd9b640 1 -- 172.21.15.146:0/407432096 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260037070 con 0x7fb27015d480 2024-04-25T15:21:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb26dd9b640 1 -- 172.21.15.146:0/407432096 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb260008640 con 0x7fb27015d480 2024-04-25T15:21:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb26dd9b640 1 -- 172.21.15.146:0/407432096 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260008a10 con 0x7fb27015d480 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 -- 172.21.15.146:0/407432096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 msgr2=0x7fb270073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 --2- 172.21.15.146:0/407432096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 0x7fb270073ae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb26000aea0 tx=0x7fb26002b740 comp rx=0 tx=0).stop 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 -- 172.21.15.146:0/407432096 shutdown_connections 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 --2- 172.21.15.146:0/407432096 >> v2:172.21.15.177:3300/0 conn(0x7fb2700b6460 0x7fb270074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 --2- 172.21.15.146:0/407432096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27015d480 0x7fb270073ae0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 -- 172.21.15.146:0/407432096 >> 172.21.15.146:0/407432096 conn(0x7fb2700d1c10 msgr2=0x7fb2700733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 -- 172.21.15.146:0/407432096 shutdown_connections 2024-04-25T15:21:05.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.369+0000 7fb275049640 1 -- 172.21.15.146:0/407432096 wait complete. 2024-04-25T15:21:05.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.371+0000 7fb275049640 1 Processor -- start 2024-04-25T15:21:05.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.372+0000 7fb275049640 1 -- start start 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb275049640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 0x7fb2701e9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb275049640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2700b6460 0x7fb27013da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb275049640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27006c550 0x7fb27006c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb275049640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb2700b5620 con 0x7fb2700ae660 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb275049640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb27006fac0 con 0x7fb27006c550 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb275049640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb2700d0f10 con 0x7fb2700b6460 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 0x7fb2701e9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 0x7fb2701e9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48846/0 (socket says 172.21.15.146:48846) 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 -- 172.21.15.146:0/2990695461 learned_addr learned my addr 172.21.15.146:0/2990695461 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26e59c640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2700b6460 0x7fb27013da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26f59e640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27006c550 0x7fb27006c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 -- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2700b6460 msgr2=0x7fb27013da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2700b6460 0x7fb27013da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 -- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27006c550 msgr2=0x7fb27006c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27006c550 0x7fb27006c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.373+0000 7fb26ed9d640 1 -- 172.21.15.146:0/2990695461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2701eaa20 con 0x7fb2700ae660 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.374+0000 7fb26ed9d640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 0x7fb2701e9a10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb260035040 tx=0x7fb260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.374+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260042020 con 0x7fb2700ae660 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.374+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb260032830 con 0x7fb2700ae660 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.374+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27006cea0 con 0x7fb2700ae660 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.374+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260037040 con 0x7fb2700ae660 2024-04-25T15:21:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.374+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2701ea460 con 0x7fb2700ae660 2024-04-25T15:21:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.375+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2700af910 con 0x7fb2700ae660 2024-04-25T15:21:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.375+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb260045020 con 0x7fb2700ae660 2024-04-25T15:21:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.376+0000 7fb2577fe640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb25c0744e0 0x7fb25c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.376+0000 7fb26e59c640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb25c0744e0 0x7fb25c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.376+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fb2600b5de0 con 0x7fb2700ae660 2024-04-25T15:21:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.377+0000 7fb26e59c640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb25c0744e0 0x7fb25c0769f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb27015c0b0 tx=0x7fb264009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.381+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb2700af910 con 0x7fb2700ae660 2024-04-25T15:21:05.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.555+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb27006a0e0 con 0x7fb2700ae660 2024-04-25T15:21:05.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.556+0000 7fb2577fe640 1 -- 172.21.15.146:0/2990695461 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7fb26004c030 con 0x7fb2700ae660 2024-04-25T15:21:05.556 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:05.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.557+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb25c0744e0 msgr2=0x7fb25c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb25c0744e0 0x7fb25c0769f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb27015c0b0 tx=0x7fb264009040 comp rx=0 tx=0).stop 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 msgr2=0x7fb2701e9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 0x7fb2701e9a10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb260035040 tx=0x7fb260004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 shutdown_connections 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb25c0744e0 0x7fb25c0769f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27006c550 0x7fb27006c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2700b6460 0x7fb27013da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 --2- 172.21.15.146:0/2990695461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2700ae660 0x7fb2701e9a10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 >> 172.21.15.146:0/2990695461 conn(0x7fb2700d1c10 msgr2=0x7fb2700b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 shutdown_connections 2024-04-25T15:21:05.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.558+0000 7fb275049640 1 -- 172.21.15.146:0/2990695461 wait complete. 2024-04-25T15:21:05.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:21:05.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.677+0000 7f258d196640 1 Processor -- start 2024-04-25T15:21:05.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.677+0000 7f258d196640 1 -- start start 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258d196640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25880b5240 0x7f25880b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258d196640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25880b5b90 con 0x7f2588162590 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258d196640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25880d1be0 con 0x7f2588161980 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258d196640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f25880d27f0 con 0x7f25880b5240 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258759e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25880b5240 0x7f25880b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2588162590 0x7f25880d2c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41730/0 (socket says 172.21.15.146:41730) 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258659c640 1 -- 172.21.15.146:0/3716969687 learned_addr learned my addr 172.21.15.146:0/3716969687 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258759e640 1 -- 172.21.15.146:0/3716969687 >> v1:172.21.15.177:6789/0 conn(0x7f2588161980 legacy=0x7f25880d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258759e640 1 -- 172.21.15.146:0/3716969687 >> v1:172.21.15.146:6789/0 conn(0x7f2588162590 legacy=0x7f25880d2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f258759e640 1 -- 172.21.15.146:0/3716969687 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25880d1ec0 con 0x7f25880b5240 2024-04-25T15:21:05.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.678+0000 7f2585d9b640 1 -- 172.21.15.146:0/3716969687 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316205353 0 0) 0x7f257c001110 con 0x7f2588162590 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.679+0000 7f258759e640 1 --2- 172.21.15.146:0/3716969687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25880b5240 0x7f25880b5650 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f25800097a0 tx=0x7f258002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb11b666a3919f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.679+0000 7f2585d9b640 1 -- 172.21.15.146:0/3716969687 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580037070 con 0x7f25880b5240 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.679+0000 7f2585d9b640 1 -- 172.21.15.146:0/3716969687 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2580032440 con 0x7f25880b5240 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.679+0000 7f2585d9b640 1 -- 172.21.15.146:0/3716969687 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580032810 con 0x7f25880b5240 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 -- 172.21.15.146:0/3716969687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25880b5240 msgr2=0x7f25880b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 --2- 172.21.15.146:0/3716969687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25880b5240 0x7f25880b5650 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f25800097a0 tx=0x7f258002a230 comp rx=0 tx=0).stop 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 -- 172.21.15.146:0/3716969687 shutdown_connections 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 --2- 172.21.15.146:0/3716969687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25880b5240 0x7f25880b5650 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 -- 172.21.15.146:0/3716969687 >> 172.21.15.146:0/3716969687 conn(0x7f25880d0240 msgr2=0x7f25880d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 -- 172.21.15.146:0/3716969687 shutdown_connections 2024-04-25T15:21:05.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.680+0000 7f258d196640 1 -- 172.21.15.146:0/3716969687 wait complete. 2024-04-25T15:21:05.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.682+0000 7f258d196640 1 Processor -- start 2024-04-25T15:21:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.682+0000 7f258d196640 1 -- start start 2024-04-25T15:21:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258d196640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25880b5240 0x7f258815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258d196640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2588161980 0x7f25881574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258d196640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 0x7f2588157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258d196640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f25880d3390 con 0x7f2588162590 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258d196640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f25880d2480 con 0x7f25880b5240 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258d196640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f25880b6c20 con 0x7f2588161980 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.683+0000 7f258759e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 0x7f2588157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 0x7f2588157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48864/0 (socket says 172.21.15.146:48864) 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 -- 172.21.15.146:0/4106816630 learned_addr learned my addr 172.21.15.146:0/4106816630 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f2586d9d640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25880b5240 0x7f258815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258659c640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2588161980 0x7f25881574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 -- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2588161980 msgr2=0x7f25881574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2588161980 0x7f25881574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 -- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25880b5240 msgr2=0x7f258815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25880b5240 0x7f258815cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 -- 172.21.15.146:0/4106816630 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2588156690 con 0x7f2588162590 2024-04-25T15:21:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258759e640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 0x7f2588157a00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f25800098c0 tx=0x7f25800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580037040 con 0x7f2588162590 2024-04-25T15:21:05.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2580042070 con 0x7f2588162590 2024-04-25T15:21:05.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2588156a00 con 0x7f2588162590 2024-04-25T15:21:05.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580030b30 con 0x7f2588162590 2024-04-25T15:21:05.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.684+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2588156cc0 con 0x7f2588162590 2024-04-25T15:21:05.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.685+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f258815b2b0 con 0x7f2588162590 2024-04-25T15:21:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.686+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2580030cd0 con 0x7f2588162590 2024-04-25T15:21:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.686+0000 7f256f7fe640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25740744e0 0x7f25740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.686+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f25800b58e0 con 0x7f2588162590 2024-04-25T15:21:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.686+0000 7f2586d9d640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25740744e0 0x7f25740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.687+0000 7f2586d9d640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25740744e0 0x7f25740769f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f25780012e0 tx=0x7f2578009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.691+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f258007fc70 con 0x7f2588162590 2024-04-25T15:21:05.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.867+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f258806a0e0 con 0x7f2588162590 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.868+0000 7f256f7fe640 1 -- 172.21.15.146:0/4106816630 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7f2580083f70 con 0x7f2588162590 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout: "poolnum": 1, 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout: "poolname": ".mgr" 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:05.869 INFO:teuthology.orchestra.run.smithi146.stdout:] 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25740744e0 msgr2=0x7f25740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25740744e0 0x7f25740769f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f25780012e0 tx=0x7f2578009040 comp rx=0 tx=0).stop 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 msgr2=0x7f2588157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 0x7f2588157a00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f25800098c0 tx=0x7f25800046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 shutdown_connections 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25740744e0 0x7f25740769f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2588162590 0x7f2588157a00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2588161980 0x7f25881574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 --2- 172.21.15.146:0/4106816630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25880b5240 0x7f258815cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 >> 172.21.15.146:0/4106816630 conn(0x7f25880d0240 msgr2=0x7f258815fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 shutdown_connections 2024-04-25T15:21:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.870+0000 7f258d196640 1 -- 172.21.15.146:0/4106816630 wait complete. 2024-04-25T15:21:05.882 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:21:05.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:21:05.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.993+0000 7fc2b839d640 1 Processor -- start 2024-04-25T15:21:05.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.993+0000 7fc2b839d640 1 -- start start 2024-04-25T15:21:05.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b839d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2b00b73e0 0x7fc2b006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b839d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc2b00d17c0 0x7fc2b00b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:05.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b839d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2b00b3450 con 0x7fc2b00b1150 2024-04-25T15:21:05.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b839d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b00d2e90 con 0x7fc2b00b73e0 2024-04-25T15:21:05.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b839d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b00b16a0 con 0x7fc2b00d17c0 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2b00b73e0 0x7fc2b006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2b00b73e0 0x7fc2b006c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48874/0 (socket says 172.21.15.146:48874) 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 -- 172.21.15.146:0/3133526638 learned_addr learned my addr 172.21.15.146:0/3133526638 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b5938640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc2b00b1150 0x7fc2b00d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41740/0 (socket says 172.21.15.146:41740) 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b693a640 1 --2- 172.21.15.146:0/3133526638 >> v2:172.21.15.177:3300/0 conn(0x7fc2b00d17c0 0x7fc2b00b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 -- 172.21.15.146:0/3133526638 >> v2:172.21.15.177:3300/0 conn(0x7fc2b00d17c0 msgr2=0x7fc2b00b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 --2- 172.21.15.146:0/3133526638 >> v2:172.21.15.177:3300/0 conn(0x7fc2b00d17c0 0x7fc2b00b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 -- 172.21.15.146:0/3133526638 >> v1:172.21.15.146:6789/0 conn(0x7fc2b00b1150 legacy=0x7fc2b00d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b6139640 1 -- 172.21.15.146:0/3133526638 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b00d2bb0 con 0x7fc2b00b73e0 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b5137640 1 -- 172.21.15.146:0/3133526638 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005508910 0 0) 0x7fc2b00b3450 con 0x7fc2b00b1150 2024-04-25T15:21:05.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.995+0000 7fc2b693a640 1 --2- 172.21.15.146:0/3133526638 >> v2:172.21.15.177:3300/0 conn(0x7fc2b00d17c0 0x7fc2b00b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.996+0000 7fc2b6139640 1 --2- 172.21.15.146:0/3133526638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00b73e0 0x7fc2b006c550 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009e50 tx=0x7fc2ac02a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295e647342b08ff9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.996+0000 7fc2b5137640 1 -- 172.21.15.146:0/3133526638 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac036070 con 0x7fc2b00b73e0 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.996+0000 7fc2b5137640 1 -- 172.21.15.146:0/3133526638 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ac031600 con 0x7fc2b00b73e0 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.996+0000 7fc2b5137640 1 -- 172.21.15.146:0/3133526638 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0319d0 con 0x7fc2b00b73e0 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 -- 172.21.15.146:0/3133526638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00b73e0 msgr2=0x7fc2b006c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3133526638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00b73e0 0x7fc2b006c550 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009e50 tx=0x7fc2ac02a760 comp rx=0 tx=0).stop 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 -- 172.21.15.146:0/3133526638 shutdown_connections 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3133526638 >> v2:172.21.15.177:3300/0 conn(0x7fc2b00d17c0 0x7fc2b00b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3133526638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00b73e0 0x7fc2b006c550 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 -- 172.21.15.146:0/3133526638 >> 172.21.15.146:0/3133526638 conn(0x7fc2b009b2b0 msgr2=0x7fc2b00b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 -- 172.21.15.146:0/3133526638 shutdown_connections 2024-04-25T15:21:05.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.997+0000 7fc2b839d640 1 -- 172.21.15.146:0/3133526638 wait complete. 2024-04-25T15:21:06.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.999+0000 7fc2b839d640 1 Processor -- start 2024-04-25T15:21:06.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:05.999+0000 7fc2b839d640 1 -- start start 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b839d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b00b1150 0x7fc2b01fa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b839d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2b00b73e0 0x7fc2b009f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b839d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 0x7fc2b01625f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b839d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0160360 con 0x7fc2b00d17c0 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b839d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b00d2880 con 0x7fc2b00b1150 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b839d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b01604e0 con 0x7fc2b00b73e0 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b693a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 0x7fc2b01625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b693a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 0x7fc2b01625f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48884/0 (socket says 172.21.15.146:48884) 2024-04-25T15:21:06.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b693a640 1 -- 172.21.15.146:0/3009695345 learned_addr learned my addr 172.21.15.146:0/3009695345 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b6139640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b00b1150 0x7fc2b01fa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.000+0000 7fc2b5938640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2b00b73e0 0x7fc2b009f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b693a640 1 -- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2b00b73e0 msgr2=0x7fc2b009f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b693a640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2b00b73e0 0x7fc2b009f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b693a640 1 -- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b00b1150 msgr2=0x7fc2b01fa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b693a640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b00b1150 0x7fc2b01fa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b693a640 1 -- 172.21.15.146:0/3009695345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b00c7830 con 0x7fc2b00d17c0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b693a640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 0x7fc2b01625f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c5d0 tx=0x7fc2a800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8015070 con 0x7fc2b00d17c0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a8004240 con 0x7fc2b00d17c0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8003610 con 0x7fc2b00d17c0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b00c7b10 con 0x7fc2b00d17c0 2024-04-25T15:21:06.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.001+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b00c80d0 con 0x7fc2b00d17c0 2024-04-25T15:21:06.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.002+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b006a0e0 con 0x7fc2b00d17c0 2024-04-25T15:21:06.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.003+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc2a80037b0 con 0x7fc2b00d17c0 2024-04-25T15:21:06.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.003+0000 7fc2a6ffd640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2940744b0 0x7fc2940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.003+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fc2a8094e50 con 0x7fc2b00d17c0 2024-04-25T15:21:06.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.003+0000 7fc2b6139640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2940744b0 0x7fc2940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.004+0000 7fc2b6139640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2940744b0 0x7fc2940769c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac033000 tx=0x7fc2ac009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:06.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.008+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2a805f7a0 con 0x7fc2b00d17c0 2024-04-25T15:21:06.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.184+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc2b0163720 con 0x7fc2b00d17c0 2024-04-25T15:21:06.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.245+0000 7fc2a6ffd640 1 -- 172.21.15.146:0/3009695345 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7fc2a805f0a0 con 0x7fc2b00d17c0 2024-04-25T15:21:06.246 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' created 2024-04-25T15:21:06.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2940744b0 msgr2=0x7fc2940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2940744b0 0x7fc2940769c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac033000 tx=0x7fc2ac009000 comp rx=0 tx=0).stop 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 msgr2=0x7fc2b01625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 0x7fc2b01625f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c5d0 tx=0x7fc2a800ca90 comp rx=0 tx=0).stop 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 shutdown_connections 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc2940744b0 0x7fc2940769c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2b00d17c0 0x7fc2b01625f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2b00b73e0 0x7fc2b009f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 --2- 172.21.15.146:0/3009695345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b00b1150 0x7fc2b01fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.247+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 >> 172.21.15.146:0/3009695345 conn(0x7fc2b009b2b0 msgr2=0x7fc2b00b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.248+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 shutdown_connections 2024-04-25T15:21:06.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.248+0000 7fc2b839d640 1 -- 172.21.15.146:0/3009695345 wait complete. 2024-04-25T15:21:06.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:21:06.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.377+0000 7fb198c92640 1 Processor -- start 2024-04-25T15:21:06.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.377+0000 7fb198c92640 1 -- start start 2024-04-25T15:21:06.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb198c92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1940b4ba0 0x7fb19406da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb198c92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19406c9c0 con 0x7fb194163be0 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb198c92640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19406cca0 con 0x7fb1940d0ef0 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb198c92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb19406d5d0 con 0x7fb1940b4ba0 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb18bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1940b4ba0 0x7fb19406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb18bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1940b4ba0 0x7fb19406da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48896/0 (socket says 172.21.15.146:48896) 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb19259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb194163be0 0x7fb1940d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41742/0 (socket says 172.21.15.146:41742) 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.378+0000 7fb18bfff640 1 -- 172.21.15.146:0/3905510888 learned_addr learned my addr 172.21.15.146:0/3905510888 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.379+0000 7fb18bfff640 1 -- 172.21.15.146:0/3905510888 >> v1:172.21.15.177:6789/0 conn(0x7fb1940d0ef0 legacy=0x7fb1940b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.379+0000 7fb18bfff640 1 -- 172.21.15.146:0/3905510888 >> v1:172.21.15.146:6789/0 conn(0x7fb194163be0 legacy=0x7fb1940d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.379+0000 7fb191d9b640 1 -- 172.21.15.146:0/3905510888 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2804929347 0 0) 0x7fb19406c9c0 con 0x7fb194163be0 2024-04-25T15:21:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.379+0000 7fb18bfff640 1 -- 172.21.15.146:0/3905510888 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19406cf80 con 0x7fb1940b4ba0 2024-04-25T15:21:06.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.379+0000 7fb18bfff640 1 --2- 172.21.15.146:0/3905510888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940b4ba0 0x7fb19406da60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009870 tx=0x7fb17c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ccddc314b2f6a6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:06.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb191d9b640 1 -- 172.21.15.146:0/3905510888 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c037070 con 0x7fb1940b4ba0 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb191d9b640 1 -- 172.21.15.146:0/3905510888 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb17c032440 con 0x7fb1940b4ba0 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb191d9b640 1 -- 172.21.15.146:0/3905510888 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c032810 con 0x7fb1940b4ba0 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 -- 172.21.15.146:0/3905510888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940b4ba0 msgr2=0x7fb19406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 --2- 172.21.15.146:0/3905510888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940b4ba0 0x7fb19406da60 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009870 tx=0x7fb17c02a2a0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 -- 172.21.15.146:0/3905510888 shutdown_connections 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 --2- 172.21.15.146:0/3905510888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940b4ba0 0x7fb19406da60 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 -- 172.21.15.146:0/3905510888 >> 172.21.15.146:0/3905510888 conn(0x7fb1940d6390 msgr2=0x7fb1940d2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 -- 172.21.15.146:0/3905510888 shutdown_connections 2024-04-25T15:21:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.380+0000 7fb198c92640 1 -- 172.21.15.146:0/3905510888 wait complete. 2024-04-25T15:21:06.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.382+0000 7fb198c92640 1 Processor -- start 2024-04-25T15:21:06.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.382+0000 7fb198c92640 1 -- start start 2024-04-25T15:21:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.383+0000 7fb198c92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1940b4ba0 0x7fb19415b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.383+0000 7fb198c92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 0x7fb19414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.383+0000 7fb198c92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb194163be0 0x7fb194161e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.383+0000 7fb198c92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb19406c550 con 0x7fb1940d0ef0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb198c92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940b4fb0 con 0x7fb1940b4ba0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb198c92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb19406c6d0 con 0x7fb194163be0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 0x7fb19414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 0x7fb19414fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48904/0 (socket says 172.21.15.146:48904) 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 -- 172.21.15.146:0/1233113194 learned_addr learned my addr 172.21.15.146:0/1233113194 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb19259c640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1940b4ba0 0x7fb19415b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb192d9d640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb194163be0 0x7fb194161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 -- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb194163be0 msgr2=0x7fb194161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb194163be0 0x7fb194161e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 -- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1940b4ba0 msgr2=0x7fb19415b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1940b4ba0 0x7fb19415b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 -- 172.21.15.146:0/1233113194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1940ca000 con 0x7fb1940d0ef0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18bfff640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 0x7fb19414fd80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009870 tx=0x7fb17c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c041070 con 0x7fb1940d0ef0 2024-04-25T15:21:06.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb17c03c870 con 0x7fb1940d0ef0 2024-04-25T15:21:06.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1940ca2e0 con 0x7fb1940d0ef0 2024-04-25T15:21:06.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c037040 con 0x7fb1940d0ef0 2024-04-25T15:21:06.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.384+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1940ca8a0 con 0x7fb1940d0ef0 2024-04-25T15:21:06.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.385+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19415bde0 con 0x7fb1940d0ef0 2024-04-25T15:21:06.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.386+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb17c044070 con 0x7fb1940d0ef0 2024-04-25T15:21:06.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.386+0000 7fb18affd640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb1740744b0 0x7fb1740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:06.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.386+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7fb17c0b5da0 con 0x7fb1940d0ef0 2024-04-25T15:21:06.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.386+0000 7fb19259c640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb1740744b0 0x7fb1740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:06.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.387+0000 7fb19259c640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb1740744b0 0x7fb1740769c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb184000f60 tx=0x7fb184007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.391+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb17c080580 con 0x7fb1940d0ef0 2024-04-25T15:21:06.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:06.570+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb1940507b0 con 0x7fb1940d0ef0 2024-04-25T15:21:07.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.247+0000 7fb18affd640 1 -- 172.21.15.146:0/1233113194 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7fb17c07fe80 con 0x7fb1940d0ef0 2024-04-25T15:21:07.248 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' created 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb1740744b0 msgr2=0x7fb1740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb1740744b0 0x7fb1740769c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb184000f60 tx=0x7fb184007c00 comp rx=0 tx=0).stop 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 msgr2=0x7fb19414fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 0x7fb19414fd80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009870 tx=0x7fb17c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 shutdown_connections 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb1740744b0 0x7fb1740769c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb194163be0 0x7fb194161e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1940d0ef0 0x7fb19414fd80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 --2- 172.21.15.146:0/1233113194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1940b4ba0 0x7fb19415b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 >> 172.21.15.146:0/1233113194 conn(0x7fb1940d6390 msgr2=0x7fb1940b9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:07.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 shutdown_connections 2024-04-25T15:21:07.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.251+0000 7fb198c92640 1 -- 172.21.15.146:0/1233113194 wait complete. 2024-04-25T15:21:07.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:21:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.389+0000 7f7717c30640 1 Processor -- start 2024-04-25T15:21:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.389+0000 7f7717c30640 1 -- start start 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.390+0000 7f7717c30640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f771008dd50 0x7f77100b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.390+0000 7f7717c30640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771008d590 con 0x7f77100b8f30 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.390+0000 7f7717c30640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77100d27d0 con 0x7f77100ce6f0 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.390+0000 7f7717c30640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f77100b9340 con 0x7f771008dd50 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.390+0000 7f77151cb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f77100b8f30 0x7f77100d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41756/0 (socket says 172.21.15.146:41756) 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.390+0000 7f77151cb640 1 -- 172.21.15.146:0/3887354950 learned_addr learned my addr 172.21.15.146:0/3887354950 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4082884302 0 0) 0x7f771008d590 con 0x7f77100b8f30 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77159cc640 1 --2- 172.21.15.146:0/3887354950 >> v2:172.21.15.177:3300/0 conn(0x7f771008dd50 0x7f77100b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76fc003520 con 0x7f77100b8f30 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1871162493 0 0) 0x7f76fc003520 con 0x7f77100b8f30 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 >> v2:172.21.15.177:3300/0 conn(0x7f771008dd50 msgr2=0x7f77100b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 --2- 172.21.15.146:0/3887354950 >> v2:172.21.15.177:3300/0 conn(0x7f771008dd50 0x7f77100b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 >> v1:172.21.15.177:6789/0 conn(0x7f77100ce6f0 legacy=0x7f77100ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77100b48b0 con 0x7f77100b8f30 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f7700001690 con 0x7f77100b8f30 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 >> v1:172.21.15.146:6789/0 conn(0x7f77100b8f30 legacy=0x7f77100d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0039b0 0x7f76fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc004800 0x7f76fc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76fc0051c0 0x7f76fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f77100b9340 con 0x7f76fc0051c0 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc00c160 con 0x7f76fc004800 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f76fc00c360 con 0x7f76fc0039b0 2024-04-25T15:21:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77161cd640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76fc0051c0 0x7f76fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77159cc640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0039b0 0x7f76fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.391+0000 7f77151cb640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc004800 0x7f76fc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 -- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0039b0 msgr2=0x7f76fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0039b0 0x7f76fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 -- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc004800 msgr2=0x7f76fc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc004800 0x7f76fc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 -- 172.21.15.146:0/3887354950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc003520 con 0x7f76fc0051c0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 -- 172.21.15.146:0/3887354950 reap_dead start 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77161cd640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76fc0051c0 0x7f76fc00b9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7708001e20 tx=0x7f76fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f009d55d6b0a4de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708001a70 con 0x7f76fc0051c0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770802ad70 con 0x7f76fc0051c0 2024-04-25T15:21:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.392+0000 7f77149ca640 1 -- 172.21.15.146:0/3887354950 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708003b90 con 0x7f76fc0051c0 2024-04-25T15:21:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 -- 172.21.15.146:0/3887354950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76fc0051c0 msgr2=0x7f76fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76fc0051c0 0x7f76fc00b9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7708001e20 tx=0x7f76fc005830 comp rx=0 tx=0).stop 2024-04-25T15:21:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 -- 172.21.15.146:0/3887354950 shutdown_connections 2024-04-25T15:21:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 --2- 172.21.15.146:0/3887354950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76fc0051c0 0x7f76fc00b9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 -- 172.21.15.146:0/3887354950 >> 172.21.15.146:0/3887354950 conn(0x7f77100b5430 msgr2=0x7f77100b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 -- 172.21.15.146:0/3887354950 shutdown_connections 2024-04-25T15:21:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.393+0000 7f7717c30640 1 -- 172.21.15.146:0/3887354950 wait complete. 2024-04-25T15:21:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.395+0000 7f7717c30640 1 Processor -- start 2024-04-25T15:21:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.395+0000 7f7717c30640 1 -- start start 2024-04-25T15:21:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f7717c30640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc0051c0 0x7f77101e6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f7717c30640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f771009e130 0x7f771009e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f7717c30640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 0x7f77101f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f7717c30640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f771008dd50 con 0x7f77101f3df0 2024-04-25T15:21:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f7717c30640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f771008d8c0 con 0x7f76fc0051c0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f7717c30640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f771008ded0 con 0x7f771009e130 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.396+0000 7f77161cd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 0x7f77101f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 0x7f77101f4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48922/0 (socket says 172.21.15.146:48922) 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 -- 172.21.15.146:0/315953791 learned_addr learned my addr 172.21.15.146:0/315953791 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77159cc640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc0051c0 0x7f77101e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77151cb640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f771009e130 0x7f771009e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 -- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f771009e130 msgr2=0x7f771009e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f771009e130 0x7f771009e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 -- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc0051c0 msgr2=0x7f77101e6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc0051c0 0x7f77101e6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 -- 172.21.15.146:0/315953791 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771009fa30 con 0x7f77101f3df0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77161cd640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 0x7f77101f4200 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f770802a820 tx=0x7f77080014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708001a70 con 0x7f77101f3df0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f770802ab70 con 0x7f77101f3df0 2024-04-25T15:21:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708005830 con 0x7f77101f3df0 2024-04-25T15:21:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771009eeb0 con 0x7f77101f3df0 2024-04-25T15:21:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.397+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771009f470 con 0x7f77101f3df0 2024-04-25T15:21:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.398+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f771009ec20 con 0x7f77101f3df0 2024-04-25T15:21:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.399+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7708032440 con 0x7f77101f3df0 2024-04-25T15:21:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.399+0000 7f77067fc640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f76f40744e0 0x7f76f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.399+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7f77080b4410 con 0x7f77101f3df0 2024-04-25T15:21:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.399+0000 7f77159cc640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f76f40744e0 0x7f76f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.400+0000 7f77159cc640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f76f40744e0 0x7f76f40769f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f77100d13f0 tx=0x7f770c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:07.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.404+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f770807eb00 con 0x7f77101f3df0 2024-04-25T15:21:07.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:07.618+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f771006a0e0 con 0x7f77101f3df0 2024-04-25T15:21:08.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.277+0000 7f77067fc640 1 -- 172.21.15.146:0/315953791 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7f770807e400 con 0x7f77101f3df0 2024-04-25T15:21:08.278 INFO:teuthology.orchestra.run.smithi146.stderr:new fs with metadata pool 5 and data pool 6 2024-04-25T15:21:08.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.289+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f76f40744e0 msgr2=0x7f76f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.289+0000 7f7717c30640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f76f40744e0 0x7f76f40769f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f77100d13f0 tx=0x7f770c008040 comp rx=0 tx=0).stop 2024-04-25T15:21:08.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.289+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 msgr2=0x7f77101f4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.289+0000 7f7717c30640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 0x7f77101f4200 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f770802a820 tx=0x7f77080014a0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.291+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 shutdown_connections 2024-04-25T15:21:08.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.291+0000 7f7717c30640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f76f40744e0 0x7f76f40769f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.291+0000 7f7717c30640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77101f3df0 0x7f77101f4200 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.291+0000 7f7717c30640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f771009e130 0x7f771009e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.291+0000 7f7717c30640 1 --2- 172.21.15.146:0/315953791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76fc0051c0 0x7f77101e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.291+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 >> 172.21.15.146:0/315953791 conn(0x7f77100b5430 msgr2=0x7f77100d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:08.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.292+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 shutdown_connections 2024-04-25T15:21:08.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.293+0000 7f7717c30640 1 -- 172.21.15.146:0/315953791 wait complete. 2024-04-25T15:21:08.312 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:08.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.422+0000 7f4a5c10d640 1 Processor -- start 2024-04-25T15:21:08.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.422+0000 7f4a5c10d640 1 -- start start 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a5c10d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a5406ee60 0x7f4a540b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a5c10d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4a54163eb0 0x7f4a541642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a5c10d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a540b8eb0 con 0x7f4a54164800 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a5c10d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4a5408d730 con 0x7f4a5406ee60 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a5c10d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f4a5406f440 con 0x7f4a54163eb0 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a5406ee60 0x7f4a540b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a5406ee60 0x7f4a540b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48938/0 (socket says 172.21.15.146:48938) 2024-04-25T15:21:08.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 -- 172.21.15.146:0/2578810278 learned_addr learned my addr 172.21.15.146:0/2578810278 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:08.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a596a8640 1 --2- 172.21.15.146:0/2578810278 >> v2:172.21.15.177:3300/0 conn(0x7f4a54163eb0 0x7f4a541642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 -- 172.21.15.146:0/2578810278 >> v2:172.21.15.177:3300/0 conn(0x7f4a54163eb0 msgr2=0x7f4a541642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:08.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 --2- 172.21.15.146:0/2578810278 >> v2:172.21.15.177:3300/0 conn(0x7f4a54163eb0 0x7f4a541642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 -- 172.21.15.146:0/2578810278 >> v1:172.21.15.177:6789/0 conn(0x7f4a54164800 legacy=0x7f4a54160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.424+0000 7f4a59ea9640 1 -- 172.21.15.146:0/2578810278 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5408ce00 con 0x7f4a5406ee60 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.425+0000 7f4a59ea9640 1 --2- 172.21.15.146:0/2578810278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5406ee60 0x7f4a540b2ba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4a540b30e0 tx=0x7f4a5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1161650bdcdcacf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.425+0000 7f4a58ea7640 1 -- 172.21.15.146:0/2578810278 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50004030 con 0x7f4a5406ee60 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.425+0000 7f4a58ea7640 1 -- 172.21.15.146:0/2578810278 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a500041d0 con 0x7f4a5406ee60 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.425+0000 7f4a58ea7640 1 -- 172.21.15.146:0/2578810278 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50003610 con 0x7f4a5406ee60 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 -- 172.21.15.146:0/2578810278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5406ee60 msgr2=0x7f4a540b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/2578810278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5406ee60 0x7f4a540b2ba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4a540b30e0 tx=0x7f4a5002a270 comp rx=0 tx=0).stop 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 -- 172.21.15.146:0/2578810278 shutdown_connections 2024-04-25T15:21:08.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/2578810278 >> v2:172.21.15.177:3300/0 conn(0x7f4a54163eb0 0x7f4a541642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/2578810278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5406ee60 0x7f4a540b2ba0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 -- 172.21.15.146:0/2578810278 >> 172.21.15.146:0/2578810278 conn(0x7f4a540b52e0 msgr2=0x7f4a540b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:08.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 -- 172.21.15.146:0/2578810278 shutdown_connections 2024-04-25T15:21:08.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.426+0000 7f4a5c10d640 1 -- 172.21.15.146:0/2578810278 wait complete. 2024-04-25T15:21:08.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.428+0000 7f4a5c10d640 1 Processor -- start 2024-04-25T15:21:08.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.428+0000 7f4a5c10d640 1 -- start start 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5c10d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a5406ee60 0x7f4a54162d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5c10d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a54163eb0 0x7f4a540a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5c10d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 0x7f4a541f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5c10d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5408dbc0 con 0x7f4a54164800 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5c10d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5408d3c0 con 0x7f4a54163eb0 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5c10d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4a540d2b70 con 0x7f4a5406ee60 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 0x7f4a541f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 0x7f4a541f6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48954/0 (socket says 172.21.15.146:48954) 2024-04-25T15:21:08.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 -- 172.21.15.146:0/927712914 learned_addr learned my addr 172.21.15.146:0/927712914 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 -- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a5406ee60 msgr2=0x7f4a54162d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a5406ee60 0x7f4a54162d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 -- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a54163eb0 msgr2=0x7f4a540a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a596a8640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a54163eb0 0x7f4a540a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a54163eb0 0x7f4a540a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 -- 172.21.15.146:0/927712914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a541604f0 con 0x7f4a54164800 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a596a8640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a54163eb0 0x7f4a540a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.429+0000 7f4a5a6aa640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 0x7f4a541f6690 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00c680 tx=0x7f4a4c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.430+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c003b70 con 0x7f4a54164800 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.430+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a4c003d10 con 0x7f4a54164800 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.430+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c005310 con 0x7f4a54164800 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.430+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a541607d0 con 0x7f4a54164800 2024-04-25T15:21:08.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.430+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a54160d90 con 0x7f4a54164800 2024-04-25T15:21:08.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.431+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a54160260 con 0x7f4a54164800 2024-04-25T15:21:08.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.431+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4a4c0054b0 con 0x7f4a54164800 2024-04-25T15:21:08.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.432+0000 7f4a4a7fc640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4a380744e0 0x7f4a380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.432+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4a4c0951a0 con 0x7f4a54164800 2024-04-25T15:21:08.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.432+0000 7f4a59ea9640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4a380744e0 0x7f4a380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.432+0000 7f4a59ea9640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4a380744e0 0x7f4a380769f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4a50033000 tx=0x7f4a5002c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:08.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.436+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a4c05f830 con 0x7f4a54164800 2024-04-25T15:21:08.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.612+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a5406a0e0 con 0x7f4a54164800 2024-04-25T15:21:08.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.613+0000 7f4a4a7fc640 1 -- 172.21.15.146:0/927712914 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+17003 (secure 0 0 0) 0x7f4a4c05f130 con 0x7f4a54164800 2024-04-25T15:21:08.614 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:08.615 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":38,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:07.619048+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:08.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4a380744e0 msgr2=0x7f4a380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4a380744e0 0x7f4a380769f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4a50033000 tx=0x7f4a5002c000 comp rx=0 tx=0).stop 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 msgr2=0x7f4a541f6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 0x7f4a541f6690 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00c680 tx=0x7f4a4c00cb40 comp rx=0 tx=0).stop 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 shutdown_connections 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4a380744e0 0x7f4a380769f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a54164800 0x7f4a541f6690 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a54163eb0 0x7f4a540a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 --2- 172.21.15.146:0/927712914 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a5406ee60 0x7f4a54162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.615+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 >> 172.21.15.146:0/927712914 conn(0x7f4a540b52e0 msgr2=0x7f4a5415e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.616+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 shutdown_connections 2024-04-25T15:21:08.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.616+0000 7f4a5c10d640 1 -- 172.21.15.146:0/927712914 wait complete. 2024-04-25T15:21:08.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:08.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.736+0000 7fd09e2e7640 1 Processor -- start 2024-04-25T15:21:08.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.736+0000 7fd09e2e7640 1 -- start start 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd09e2e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 0x7fd0980d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd09e2e7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd098160680 con 0x7fd09808d370 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd09e2e7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0980d20e0 con 0x7fd09806ee60 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd09e2e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd0980d23c0 con 0x7fd0980b2df0 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd09c884640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd09808d370 0x7fd0980cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41760/0 (socket says 172.21.15.146:41760) 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd09c884640 1 -- 172.21.15.146:0/2429281085 learned_addr learned my addr 172.21.15.146:0/2429281085 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.737+0000 7fd097fff640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 0x7fd0980d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 738623154 0 0) 0x7fd098160680 con 0x7fd09808d370 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd080003520 con 0x7fd09808d370 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3610171385 0 0) 0x7fd080003520 con 0x7fd09808d370 2024-04-25T15:21:08.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 msgr2=0x7fd0980d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 0x7fd0980d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 >> v1:172.21.15.177:6789/0 conn(0x7fd09806ee60 legacy=0x7fd09808cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0980d26a0 con 0x7fd09808d370 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fd0900019f0 con 0x7fd09808d370 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 >> v1:172.21.15.146:6789/0 conn(0x7fd09808d370 legacy=0x7fd0980cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 0x7fd0800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd080004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd080004d40 0x7fd080005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980d23c0 con 0x7fd080003f40 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd08000bb80 con 0x7fd080004d40 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd08000bd80 con 0x7fd0980b2df0 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd080004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd097fff640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 0x7fd0800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd09c884640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd080004d40 0x7fd080005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd0977fe640 1 -- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 msgr2=0x7fd0800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0980b2df0 0x7fd0800039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd0977fe640 1 -- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd080004d40 msgr2=0x7fd080005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd080004d40 0x7fd080005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.738+0000 7fd0977fe640 1 -- 172.21.15.146:0/2429281085 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd080003520 con 0x7fd080003f40 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.739+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd080004800 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd08c001860 tx=0x7fd080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdd7a66adaa8339 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.739+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c037070 con 0x7fd080003f40 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.739+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08c02abf0 con 0x7fd080003f40 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.739+0000 7fd096ffd640 1 -- 172.21.15.146:0/2429281085 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c001430 con 0x7fd080003f40 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2429281085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 msgr2=0x7fd080004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2429281085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd080004800 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd08c001860 tx=0x7fd080005830 comp rx=0 tx=0).stop 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09c884640 1 -- 172.21.15.146:0/2429281085 reap_dead start 2024-04-25T15:21:08.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2429281085 shutdown_connections 2024-04-25T15:21:08.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2429281085 >> 172.21.15.146:0/2429281085 conn(0x7fd0980b5360 msgr2=0x7fd0980b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:08.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2429281085 shutdown_connections 2024-04-25T15:21:08.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.740+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2429281085 wait complete. 2024-04-25T15:21:08.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.742+0000 7fd09e2e7640 1 Processor -- start 2024-04-25T15:21:08.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.742+0000 7fd09e2e7640 1 -- start start 2024-04-25T15:21:08.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09e2e7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd09815f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09e2e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd098161a30 0x7fd098161e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09e2e7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0980c8660 0x7fd0980c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09e2e7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980b91e0 con 0x7fd080003f40 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09e2e7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980d2c60 con 0x7fd0980c8660 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09e2e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd0980d1c70 con 0x7fd098161a30 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd097fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd09815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd097fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd09815f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48968/0 (socket says 172.21.15.146:48968) 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd097fff640 1 -- 172.21.15.146:0/2631557951 learned_addr learned my addr 172.21.15.146:0/2631557951 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd098161a30 0x7fd098161e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.743+0000 7fd09c884640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0980c8660 0x7fd0980c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd097fff640 1 -- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd098161a30 msgr2=0x7fd098161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd097fff640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd098161a30 0x7fd098161e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd097fff640 1 -- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0980c8660 msgr2=0x7fd0980c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd097fff640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0980c8660 0x7fd0980c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd097fff640 1 -- 172.21.15.146:0/2631557951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd098162380 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd09c884640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0980c8660 0x7fd0980c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd097fff640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd09815f770 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd088009fd0 tx=0x7fd088011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800faf0 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08800fc90 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0980c7ea0 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088014440 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.744+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0980b8210 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.745+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0980b2100 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.745+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd088012040 con 0x7fd080003f40 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.746+0000 7fd094ff9640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0840744b0 0x7fd0840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.746+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0840744b0 0x7fd0840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:08.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.746+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd088095a30 con 0x7fd080003f40 2024-04-25T15:21:08.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.746+0000 7fd0977fe640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0840744b0 0x7fd0840769c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd08c034000 tx=0x7fd08c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:08.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.751+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0980b2100 con 0x7fd080003f40 2024-04-25T15:21:08.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.925+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd09806a0e0 con 0x7fd080003f40 2024-04-25T15:21:08.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.926+0000 7fd094ff9640 1 -- 172.21.15.146:0/2631557951 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+17003 (secure 0 0 0) 0x7fd098162380 con 0x7fd080003f40 2024-04-25T15:21:08.927 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:08.927 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":38,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:07.619048+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:08.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0840744b0 msgr2=0x7fd0840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0840744b0 0x7fd0840769c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd08c034000 tx=0x7fd08c009030 comp rx=0 tx=0).stop 2024-04-25T15:21:08.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 msgr2=0x7fd09815f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd09815f770 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd088009fd0 tx=0x7fd088011dc0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 shutdown_connections 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd0840744b0 0x7fd0840769c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0980c8660 0x7fd0980c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd098161a30 0x7fd098161e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 --2- 172.21.15.146:0/2631557951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd080003f40 0x7fd09815f770 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.928+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 >> 172.21.15.146:0/2631557951 conn(0x7fd0980b5360 msgr2=0x7fd09815e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.929+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 shutdown_connections 2024-04-25T15:21:08.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:08.929+0000 7fd09e2e7640 1 -- 172.21.15.146:0/2631557951 wait complete. 2024-04-25T15:21:08.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:21:09.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.051+0000 7f3d24c42640 1 Processor -- start 2024-04-25T15:21:09.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.051+0000 7f3d24c42640 1 -- start start 2024-04-25T15:21:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d24c42640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3d20163c00 0x7f3d2006d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d24c42640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 0x7f3d200d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d24c42640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2006dea0 con 0x7f3d200d0de0 2024-04-25T15:21:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d24c42640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3d200b5230 con 0x7f3d20163c00 2024-04-25T15:21:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d24c42640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2006c8c0 con 0x7f3d200bb450 2024-04-25T15:21:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d1e59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3d20163c00 0x7f3d2006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d1dd9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 0x7f3d200d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d1e59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3d20163c00 0x7f3d2006d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:37828/0 (socket says 172.21.15.146:37828) 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d1dd9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 0x7f3d200d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:42222/0 (socket says 172.21.15.146:42222) 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.053+0000 7f3d1e59c640 1 -- 172.21.15.146:0/1110819503 learned_addr learned my addr 172.21.15.146:0/1110819503 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1e59c640 1 -- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 msgr2=0x7f3d200d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1e59c640 1 --2- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 0x7f3d200d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1e59c640 1 -- 172.21.15.146:0/1110819503 >> v1:172.21.15.177:6789/0 conn(0x7f3d200d0de0 legacy=0x7f3d200b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1e59c640 1 -- 172.21.15.146:0/1110819503 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2006cba0 con 0x7f3d20163c00 2024-04-25T15:21:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1dd9b640 1 --2- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 0x7f3d200d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:09.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1e59c640 1 --2- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d20163c00 0x7f3d2006d960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009e50 tx=0x7f3d1002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d0de54ca25b1f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:09.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.054+0000 7f3d1d59a640 1 -- 172.21.15.146:0/1110819503 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10036070 con 0x7f3d20163c00 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d1d59a640 1 -- 172.21.15.146:0/1110819503 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d10031600 con 0x7f3d20163c00 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d1d59a640 1 -- 172.21.15.146:0/1110819503 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d100319d0 con 0x7f3d20163c00 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 -- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d20163c00 msgr2=0x7f3d2006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 --2- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d20163c00 0x7f3d2006d960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009e50 tx=0x7f3d1002a680 comp rx=0 tx=0).stop 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 -- 172.21.15.146:0/1110819503 shutdown_connections 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 --2- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d200bb450 0x7f3d200d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 --2- 172.21.15.146:0/1110819503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d20163c00 0x7f3d2006d960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 -- 172.21.15.146:0/1110819503 >> 172.21.15.146:0/1110819503 conn(0x7f3d200b7970 msgr2=0x7f3d200b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 -- 172.21.15.146:0/1110819503 shutdown_connections 2024-04-25T15:21:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.055+0000 7f3d24c42640 1 -- 172.21.15.146:0/1110819503 wait complete. 2024-04-25T15:21:09.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.057+0000 7f3d24c42640 1 Processor -- start 2024-04-25T15:21:09.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.057+0000 7f3d24c42640 1 -- start start 2024-04-25T15:21:09.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d24c42640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 0x7f3d2014ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:09.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d24c42640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d200d0de0 0x7f3d200cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:09.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d24c42640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d20163c00 0x7f3d20151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d24c42640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2006c550 con 0x7f3d200bb450 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d24c42640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2006d160 con 0x7f3d200d0de0 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d24c42640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3d200bb860 con 0x7f3d20163c00 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.058+0000 7f3d1e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 0x7f3d2014ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 0x7f3d2014ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48980/0 (socket says 172.21.15.146:48980) 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 -- 172.21.15.146:0/2042030893 learned_addr learned my addr 172.21.15.146:0/2042030893 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1ed9d640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d20163c00 0x7f3d20151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1dd9b640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d200d0de0 0x7f3d200cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 -- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d20163c00 msgr2=0x7f3d20151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d20163c00 0x7f3d20151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 -- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d200d0de0 msgr2=0x7f3d200cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d200d0de0 0x7f3d200cc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 -- 172.21.15.146:0/2042030893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d200cb5c0 con 0x7f3d200bb450 2024-04-25T15:21:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d1e59c640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 0x7f3d2014ead0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009e50 tx=0x7f3d10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10036050 con 0x7f3d200bb450 2024-04-25T15:21:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d200cb8a0 con 0x7f3d200bb450 2024-04-25T15:21:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d10042070 con 0x7f3d200bb450 2024-04-25T15:21:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10009510 con 0x7f3d200bb450 2024-04-25T15:21:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.059+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d20150c80 con 0x7f3d200bb450 2024-04-25T15:21:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.060+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d20150ee0 con 0x7f3d200bb450 2024-04-25T15:21:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.061+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d10007c70 con 0x7f3d200bb450 2024-04-25T15:21:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.061+0000 7f3d0effd640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3d000744e0 0x7f3d000769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.061+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3d100b6100 con 0x7f3d200bb450 2024-04-25T15:21:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.062+0000 7f3d1dd9b640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3d000744e0 0x7f3d000769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.062+0000 7f3d1dd9b640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3d000744e0 0x7f3d000769f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d200cf240 tx=0x7f3d14009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:09.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.066+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d10080030 con 0x7f3d200bb450 2024-04-25T15:21:09.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:09.285+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3d2006a0e0 con 0x7f3d200bb450 2024-04-25T15:21:10.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.290+0000 7f3d0effd640 1 -- 172.21.15.146:0/2042030893 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f3d10084330 con 0x7f3d200bb450 2024-04-25T15:21:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3d000744e0 msgr2=0x7f3d000769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3d000744e0 0x7f3d000769f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d200cf240 tx=0x7f3d14009190 comp rx=0 tx=0).stop 2024-04-25T15:21:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 msgr2=0x7f3d2014ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 0x7f3d2014ead0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009e50 tx=0x7f3d10004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:10.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 shutdown_connections 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3d000744e0 0x7f3d000769f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d20163c00 0x7f3d20151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d200d0de0 0x7f3d200cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 --2- 172.21.15.146:0/2042030893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d200bb450 0x7f3d2014ead0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 >> 172.21.15.146:0/2042030893 conn(0x7f3d200b7970 msgr2=0x7f3d20162780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 shutdown_connections 2024-04-25T15:21:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.293+0000 7f3d24c42640 1 -- 172.21.15.146:0/2042030893 wait complete. 2024-04-25T15:21:10.307 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:21:10.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:21:10.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.425+0000 7ff9ad005640 1 Processor -- start 2024-04-25T15:21:10.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.425+0000 7ff9ad005640 1 -- start start 2024-04-25T15:21:10.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.426+0000 7ff9ad005640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9a8160210 0x7ff9a809b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.426+0000 7ff9ad005640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9a806d8d0 0x7ff9a80af130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.426+0000 7ff9ad005640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a80af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.426+0000 7ff9ad005640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a806d440 con 0x7ff9a806d8d0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.426+0000 7ff9ad005640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a806c550 con 0x7ff9a8160210 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.426+0000 7ff9ad005640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a806c830 con 0x7ff9a806dce0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9a806d8d0 0x7ff9a80af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9a806d8d0 0x7ff9a80af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48986/0 (socket says 172.21.15.146:48986) 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 -- 172.21.15.146:0/757630139 learned_addr learned my addr 172.21.15.146:0/757630139 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a6d9d640 1 --2- 172.21.15.146:0/757630139 >> v2:172.21.15.177:3300/0 conn(0x7ff9a8160210 0x7ff9a809b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a759e640 1 --2- 172.21.15.146:0/757630139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a80af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 -- 172.21.15.146:0/757630139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 msgr2=0x7ff9a80af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 --2- 172.21.15.146:0/757630139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a80af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 -- 172.21.15.146:0/757630139 >> v2:172.21.15.177:3300/0 conn(0x7ff9a8160210 msgr2=0x7ff9a809b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 --2- 172.21.15.146:0/757630139 >> v2:172.21.15.177:3300/0 conn(0x7ff9a8160210 0x7ff9a809b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.427+0000 7ff9a659c640 1 -- 172.21.15.146:0/757630139 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a806cb10 con 0x7ff9a806d8d0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9a659c640 1 --2- 172.21.15.146:0/757630139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a80af130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8072270 tx=0x7ff99c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a105ddb38c27a2bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9a5d9b640 1 -- 172.21.15.146:0/757630139 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c037070 con 0x7ff9a806d8d0 2024-04-25T15:21:10.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9a5d9b640 1 -- 172.21.15.146:0/757630139 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff99c032440 con 0x7ff9a806d8d0 2024-04-25T15:21:10.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9a5d9b640 1 -- 172.21.15.146:0/757630139 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c032810 con 0x7ff9a806d8d0 2024-04-25T15:21:10.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 -- 172.21.15.146:0/757630139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 msgr2=0x7ff9a80af130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:10.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 --2- 172.21.15.146:0/757630139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a80af130 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8072270 tx=0x7ff99c02a140 comp rx=0 tx=0).stop 2024-04-25T15:21:10.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 -- 172.21.15.146:0/757630139 shutdown_connections 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 --2- 172.21.15.146:0/757630139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a80af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 --2- 172.21.15.146:0/757630139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a80af130 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 --2- 172.21.15.146:0/757630139 >> v2:172.21.15.177:3300/0 conn(0x7ff9a8160210 0x7ff9a809b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.428+0000 7ff9ad005640 1 -- 172.21.15.146:0/757630139 >> 172.21.15.146:0/757630139 conn(0x7ff9a8073a30 msgr2=0x7ff9a80b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.429+0000 7ff9ad005640 1 -- 172.21.15.146:0/757630139 shutdown_connections 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.429+0000 7ff9ad005640 1 -- 172.21.15.146:0/757630139 wait complete. 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.431+0000 7ff9ad005640 1 Processor -- start 2024-04-25T15:21:10.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.431+0000 7ff9ad005640 1 -- start start 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9ad005640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a81ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9ad005640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a8161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9ad005640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a8160210 0x7ff9a8162300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9ad005640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a80afbb0 con 0x7ff9a806d8d0 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9ad005640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a806d0d0 con 0x7ff9a8160210 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9ad005640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a80afd30 con 0x7ff9a806dce0 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a81ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a81ebff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49000/0 (socket says 172.21.15.146:49000) 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 -- 172.21.15.146:0/1562744443 learned_addr learned my addr 172.21.15.146:0/1562744443 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:10.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a659c640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a8161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a759e640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a8160210 0x7ff9a8162300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 -- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 msgr2=0x7ff9a8161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a8161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 -- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a8160210 msgr2=0x7ff9a8162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a8160210 0x7ff9a8162300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.432+0000 7ff9a6d9d640 1 -- 172.21.15.146:0/1562744443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a815c130 con 0x7ff9a806d8d0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.433+0000 7ff9a6d9d640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a81ebff0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff99800d5d0 tx=0x7ff99800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.433+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998004240 con 0x7ff9a806d8d0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.433+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9980043e0 con 0x7ff9a806d8d0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.433+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a815c410 con 0x7ff9a806d8d0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.433+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998003830 con 0x7ff9a806d8d0 2024-04-25T15:21:10.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.433+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a815cc40 con 0x7ff9a806d8d0 2024-04-25T15:21:10.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.434+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a806a0e0 con 0x7ff9a806d8d0 2024-04-25T15:21:10.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.434+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff9980039d0 con 0x7ff9a806d8d0 2024-04-25T15:21:10.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.435+0000 7ff98f7fe640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9940744b0 0x7ff9940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:10.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.435+0000 7ff9a659c640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9940744b0 0x7ff9940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:10.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.435+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff998095610 con 0x7ff9a806d8d0 2024-04-25T15:21:10.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.436+0000 7ff9a659c640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9940744b0 0x7ff9940769c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff99c034000 tx=0x7ff99c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:10.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.440+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff99805fca0 con 0x7ff9a806d8d0 2024-04-25T15:21:10.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:10.650+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff9a815c5e0 con 0x7ff9a806d8d0 2024-04-25T15:21:11.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.292+0000 7ff98f7fe640 1 -- 172.21.15.146:0/1562744443 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7ff99805f5a0 con 0x7ff9a806d8d0 2024-04-25T15:21:11.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9940744b0 msgr2=0x7ff9940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9940744b0 0x7ff9940769c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff99c034000 tx=0x7ff99c02e000 comp rx=0 tx=0).stop 2024-04-25T15:21:11.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 msgr2=0x7ff9a81ebff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a81ebff0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff99800d5d0 tx=0x7ff99800da90 comp rx=0 tx=0).stop 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 shutdown_connections 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9940744b0 0x7ff9940769c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a8160210 0x7ff9a8162300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a806dce0 0x7ff9a8161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 --2- 172.21.15.146:0/1562744443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a806d8d0 0x7ff9a81ebff0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 >> 172.21.15.146:0/1562744443 conn(0x7ff9a8073a30 msgr2=0x7ff9a815b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 shutdown_connections 2024-04-25T15:21:11.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.294+0000 7ff9ad005640 1 -- 172.21.15.146:0/1562744443 wait complete. 2024-04-25T15:21:11.307 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:21:11.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:21:11.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.417+0000 7ff3813e3640 1 Processor -- start 2024-04-25T15:21:11.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.417+0000 7ff3813e3640 1 -- start start 2024-04-25T15:21:11.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff3813e3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff37c0d4f60 0x7ff37c0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff3813e3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c0b9200 con 0x7ff37c06ccc0 2024-04-25T15:21:11.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff3813e3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff37c0d1070 con 0x7ff37c0d0020 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff3813e3640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff37c06c550 con 0x7ff37c0d4f60 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff37b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff37c06ccc0 0x7ff37c06d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54286/0 (socket says 172.21.15.146:54286) 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff37b7fe640 1 -- 172.21.15.146:0/1477590455 learned_addr learned my addr 172.21.15.146:0/1477590455 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff37affd640 1 --2- 172.21.15.146:0/1477590455 >> v2:172.21.15.177:3300/0 conn(0x7ff37c0d4f60 0x7ff37c0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2056674088 0 0) 0x7ff374001110 con 0x7ff37c06ccc0 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff364003520 con 0x7ff37c06ccc0 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1439220423 0 0) 0x7ff364003520 con 0x7ff37c06ccc0 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 >> v2:172.21.15.177:3300/0 conn(0x7ff37c0d4f60 msgr2=0x7ff37c0d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 --2- 172.21.15.146:0/1477590455 >> v2:172.21.15.177:3300/0 conn(0x7ff37c0d4f60 0x7ff37c0d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 >> v1:172.21.15.177:6789/0 conn(0x7ff37c0d0020 legacy=0x7ff37c0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c0d1310 con 0x7ff37c06ccc0 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7ff374001840 con 0x7ff37c06ccc0 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 >> v1:172.21.15.146:6789/0 conn(0x7ff37c06ccc0 legacy=0x7ff37c06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3640039b0 0x7ff364003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff364004800 0x7ff364004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff364005350 0x7ff36400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c06c550 con 0x7ff3640039b0 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff36400c280 con 0x7ff364005350 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff36400c480 con 0x7ff364004800 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.419+0000 7ff37affd640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3640039b0 0x7ff364003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37b7fe640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff364005350 0x7ff36400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37a7fc640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff364004800 0x7ff364004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37affd640 1 -- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff364004800 msgr2=0x7ff364004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37affd640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff364004800 0x7ff364004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37affd640 1 -- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff364005350 msgr2=0x7ff36400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37affd640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff364005350 0x7ff36400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37affd640 1 -- 172.21.15.146:0/1477590455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff364003520 con 0x7ff3640039b0 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37b7fe640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff364005350 0x7ff36400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37b7fe640 1 -- 172.21.15.146:0/1477590455 reap_dead start 2024-04-25T15:21:11.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.420+0000 7ff37affd640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3640039b0 0x7ff364003e20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff37c163ce0 tx=0x7ff364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38bcf7df19a39c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c037070 con 0x7ff3640039b0 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36c02e050 con 0x7ff3640039b0 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff379ffb640 1 -- 172.21.15.146:0/1477590455 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c009c10 con 0x7ff3640039b0 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 -- 172.21.15.146:0/1477590455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3640039b0 msgr2=0x7ff364003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3640039b0 0x7ff364003e20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff37c163ce0 tx=0x7ff364005830 comp rx=0 tx=0).stop 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 -- 172.21.15.146:0/1477590455 shutdown_connections 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 --2- 172.21.15.146:0/1477590455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3640039b0 0x7ff364003e20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 -- 172.21.15.146:0/1477590455 >> 172.21.15.146:0/1477590455 conn(0x7ff37c0d2450 msgr2=0x7ff37c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 -- 172.21.15.146:0/1477590455 shutdown_connections 2024-04-25T15:21:11.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.421+0000 7ff3813e3640 1 -- 172.21.15.146:0/1477590455 wait complete. 2024-04-25T15:21:11.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.423+0000 7ff3813e3640 1 Processor -- start 2024-04-25T15:21:11.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.423+0000 7ff3813e3640 1 -- start start 2024-04-25T15:21:11.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.424+0000 7ff3813e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff3640039b0 0x7ff37c1e6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.424+0000 7ff3813e3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff37c1e7da0 0x7ff37c148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.424+0000 7ff3813e3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 0x7ff37c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.424+0000 7ff3813e3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c0b5150 con 0x7ff37c149320 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.424+0000 7ff3813e3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c06c830 con 0x7ff37c1e7da0 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.424+0000 7ff3813e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff37c0b52d0 con 0x7ff3640039b0 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 0x7ff37c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 0x7ff37c0c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38088/0 (socket says 172.21.15.146:38088) 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 -- 172.21.15.146:0/2671697796 learned_addr learned my addr 172.21.15.146:0/2671697796 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37affd640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff3640039b0 0x7ff37c1e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37a7fc640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff37c1e7da0 0x7ff37c148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 -- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff3640039b0 msgr2=0x7ff37c1e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff3640039b0 0x7ff37c1e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 -- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff37c1e7da0 msgr2=0x7ff37c148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff37c1e7da0 0x7ff37c148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 -- 172.21.15.146:0/2671697796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c149730 con 0x7ff37c149320 2024-04-25T15:21:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff37b7fe640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 0x7ff37c0c9620 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0d0eb0 tx=0x7ff374007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400e610 con 0x7ff37c149320 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff374018070 con 0x7ff37c149320 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c1e6d00 con 0x7ff37c149320 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400e9f0 con 0x7ff37c149320 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.425+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c1e72c0 con 0x7ff37c149320 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.426+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37c06a0e0 con 0x7ff37c149320 2024-04-25T15:21:11.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.427+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff37400eb90 con 0x7ff37c149320 2024-04-25T15:21:11.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.427+0000 7ff3637fe640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff3680744b0 0x7ff3680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.427+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff374094e70 con 0x7ff37c149320 2024-04-25T15:21:11.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.427+0000 7ff37affd640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff3680744b0 0x7ff3680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.428+0000 7ff37affd640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff3680744b0 0x7ff3680769c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff36c034000 tx=0x7ff36c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:11.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.432+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff37c0b9200 con 0x7ff37c149320 2024-04-25T15:21:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.646+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff37c1e6a90 con 0x7ff37c149320 2024-04-25T15:21:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.646+0000 7ff3637fe640 1 -- 172.21.15.146:0/2671697796 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5187 (secure 0 0 0) 0x7ff37405ee20 con 0x7ff37c149320 2024-04-25T15:21:11.647 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:11.647 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:10.651740+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5556":{"gid":5556,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6835/539736758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":539736758},{"type":"v1","addr":"172.21.15.177:6835","nonce":539736758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5556,"qdb_cluster":[5556]},"id":2}]} 2024-04-25T15:21:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 17 2024-04-25T15:21:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff3680744b0 msgr2=0x7ff3680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff3680744b0 0x7ff3680769c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff36c034000 tx=0x7ff36c02f000 comp rx=0 tx=0).stop 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 msgr2=0x7ff37c0c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 0x7ff37c0c9620 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0d0eb0 tx=0x7ff374007420 comp rx=0 tx=0).stop 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 shutdown_connections 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff3680744b0 0x7ff3680769c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c149320 0x7ff37c0c9620 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff37c1e7da0 0x7ff37c148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 --2- 172.21.15.146:0/2671697796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff3640039b0 0x7ff37c1e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 >> 172.21.15.146:0/2671697796 conn(0x7ff37c0d2450 msgr2=0x7ff37c0b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 shutdown_connections 2024-04-25T15:21:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.648+0000 7ff3813e3640 1 -- 172.21.15.146:0/2671697796 wait complete. 2024-04-25T15:21:11.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.768+0000 7f2746af9640 1 Processor -- start 2024-04-25T15:21:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.768+0000 7f2746af9640 1 -- start start 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2746af9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2746af9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2740161b30 0x7f2740161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2746af9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27400d1420 con 0x7f27400cf1d0 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2746af9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f27400ced40 con 0x7f2740161b30 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2746af9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f27400d1700 con 0x7f27400d1cc0 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2744895640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f27400cf1d0 0x7f27400b7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54294/0 (socket says 172.21.15.146:54294) 2024-04-25T15:21:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2744895640 1 -- 172.21.15.146:0/2573051626 learned_addr learned my addr 172.21.15.146:0/2573051626 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2745096640 1 --2- 172.21.15.146:0/2573051626 >> v2:172.21.15.177:3300/0 conn(0x7f2740161b30 0x7f2740161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f273ffff640 1 --2- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f273f7fe640 1 -- 172.21.15.146:0/2573051626 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2455677310 0 0) 0x7f27400d1420 con 0x7f27400cf1d0 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f273f7fe640 1 -- 172.21.15.146:0/2573051626 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2728003520 con 0x7f27400cf1d0 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2745096640 1 -- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 msgr2=0x7f27400d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2745096640 1 --2- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400d20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2745096640 1 -- 172.21.15.146:0/2573051626 >> v1:172.21.15.146:6789/0 conn(0x7f27400cf1d0 legacy=0x7f27400b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.770+0000 7f2745096640 1 -- 172.21.15.146:0/2573051626 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27400d19e0 con 0x7f2740161b30 2024-04-25T15:21:11.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.771+0000 7f2745096640 1 --2- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2740161b30 0x7f2740161f40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2738009d90 tx=0x7f273802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a63349ec49ba437 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:11.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.771+0000 7f273f7fe640 1 -- 172.21.15.146:0/2573051626 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738007740 con 0x7f2740161b30 2024-04-25T15:21:11.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.771+0000 7f273f7fe640 1 -- 172.21.15.146:0/2573051626 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27380078e0 con 0x7f2740161b30 2024-04-25T15:21:11.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.771+0000 7f273f7fe640 1 -- 172.21.15.146:0/2573051626 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738007c90 con 0x7f2740161b30 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 -- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2740161b30 msgr2=0x7f2740161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 --2- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2740161b30 0x7f2740161f40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2738009d90 tx=0x7f273802a660 comp rx=0 tx=0).stop 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 -- 172.21.15.146:0/2573051626 shutdown_connections 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 --2- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2740161b30 0x7f2740161f40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 --2- 172.21.15.146:0/2573051626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 -- 172.21.15.146:0/2573051626 >> 172.21.15.146:0/2573051626 conn(0x7f274009b2b0 msgr2=0x7f27400b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 -- 172.21.15.146:0/2573051626 shutdown_connections 2024-04-25T15:21:11.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.772+0000 7f2746af9640 1 -- 172.21.15.146:0/2573051626 wait complete. 2024-04-25T15:21:11.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.774+0000 7f2746af9640 1 Processor -- start 2024-04-25T15:21:11.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.774+0000 7f2746af9640 1 -- start start 2024-04-25T15:21:11.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2746af9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27400cf1d0 0x7f27401573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2746af9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2746af9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 0x7f27400cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2746af9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f27400d2610 con 0x7f2740161b30 2024-04-25T15:21:11.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2746af9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f27400cf5e0 con 0x7f27400cf1d0 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2746af9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f274015e310 con 0x7f27400d1cc0 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.775+0000 7f2745096640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 0x7f27400cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 0x7f27400cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38096/0 (socket says 172.21.15.146:38096) 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 -- 172.21.15.146:0/1334377984 learned_addr learned my addr 172.21.15.146:0/1334377984 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2744895640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27400cf1d0 0x7f27401573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f273ffff640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 -- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 msgr2=0x7f27400cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 -- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27400cf1d0 msgr2=0x7f27401573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27400cf1d0 0x7f27401573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 -- 172.21.15.146:0/1334377984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27400cb8c0 con 0x7f2740161b30 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f273ffff640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2745096640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 0x7f27400cb0b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f273802ac50 tx=0x7f2738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273803f070 con 0x7f2740161b30 2024-04-25T15:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2738031c60 con 0x7f2740161b30 2024-04-25T15:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738009450 con 0x7f2740161b30 2024-04-25T15:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27400c9f90 con 0x7f2740161b30 2024-04-25T15:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.776+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27400c5df0 con 0x7f2740161b30 2024-04-25T15:21:11.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.777+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f274006a0e0 con 0x7f2740161b30 2024-04-25T15:21:11.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.778+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f273803b410 con 0x7f2740161b30 2024-04-25T15:21:11.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.778+0000 7f273d7fa640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f272c0744b0 0x7f272c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:11.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.778+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f27380b66d0 con 0x7f2740161b30 2024-04-25T15:21:11.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.778+0000 7f2744895640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f272c0744b0 0x7f272c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:11.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.779+0000 7f2744895640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f272c0744b0 0x7f272c0769c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2730005880 tx=0x7f2730009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:11.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.783+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2738080cd0 con 0x7f2740161b30 2024-04-25T15:21:11.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.957+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27400c9d20 con 0x7f2740161b30 2024-04-25T15:21:11.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.959+0000 7f273d7fa640 1 -- 172.21.15.146:0/1334377984 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16990 (secure 0 0 0) 0x7f27380805d0 con 0x7f2740161b30 2024-04-25T15:21:11.960 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:11.960 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":39,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:09.275980+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f272c0744b0 msgr2=0x7f272c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f272c0744b0 0x7f272c0769c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2730005880 tx=0x7f2730009040 comp rx=0 tx=0).stop 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 msgr2=0x7f27400cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 0x7f27400cb0b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f273802ac50 tx=0x7f2738004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 shutdown_connections 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f272c0744b0 0x7f272c0769c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2740161b30 0x7f27400cb0b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27400d1cc0 0x7f27400cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 --2- 172.21.15.146:0/1334377984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27400cf1d0 0x7f27401573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 >> 172.21.15.146:0/1334377984 conn(0x7f274009b2b0 msgr2=0x7f274006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:11.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 shutdown_connections 2024-04-25T15:21:11.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:11.961+0000 7f2746af9640 1 -- 172.21.15.146:0/1334377984 wait complete. 2024-04-25T15:21:11.973 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:21:11.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:21:12.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.081+0000 7efe7a6f9640 1 Processor -- start 2024-04-25T15:21:12.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.081+0000 7efe7a6f9640 1 -- start start 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.082+0000 7efe7a6f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe740b6610 0x7efe740d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.082+0000 7efe7a6f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efe7415ffa0 0x7efe740b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.082+0000 7efe7a6f9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe740b5b90 con 0x7efe740b5780 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.082+0000 7efe7a6f9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7efe740b2f50 con 0x7efe7415ffa0 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.082+0000 7efe7a6f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efe740b2ac0 con 0x7efe740b6610 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efe7415ffa0 0x7efe740b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe73fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe740b6610 0x7efe740d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efe7415ffa0 0x7efe740b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56124/0 (socket says 172.21.15.146:56124) 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 -- 172.21.15.146:0/1043537959 learned_addr learned my addr 172.21.15.146:0/1043537959 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe78c96640 1 --1- 172.21.15.146:0/1043537959 >> v1:172.21.15.177:6789/0 conn(0x7efe740b5780 0x7efe74163b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:58818/0 (socket says 172.21.15.146:58818) 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 -- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe740b6610 msgr2=0x7efe740d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 --2- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe740b6610 0x7efe740d2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 -- 172.21.15.146:0/1043537959 >> v1:172.21.15.177:6789/0 conn(0x7efe740b5780 legacy=0x7efe74163b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.083+0000 7efe737fe640 1 -- 172.21.15.146:0/1043537959 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe740d2230 con 0x7efe7415ffa0 2024-04-25T15:21:12.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.084+0000 7efe737fe640 1 --2- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe7415ffa0 0x7efe740b5240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efe68009870 tx=0x7efe6802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecf6630113e5e67d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.084+0000 7efe72ffd640 1 -- 172.21.15.146:0/1043537959 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68037070 con 0x7efe7415ffa0 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.084+0000 7efe72ffd640 1 -- 172.21.15.146:0/1043537959 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe68032440 con 0x7efe7415ffa0 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.084+0000 7efe72ffd640 1 -- 172.21.15.146:0/1043537959 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68032810 con 0x7efe7415ffa0 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 -- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe7415ffa0 msgr2=0x7efe740b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe7415ffa0 0x7efe740b5240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efe68009870 tx=0x7efe6802a250 comp rx=0 tx=0).stop 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 -- 172.21.15.146:0/1043537959 shutdown_connections 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe7415ffa0 0x7efe740b5240 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/1043537959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe740b6610 0x7efe740d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 -- 172.21.15.146:0/1043537959 >> 172.21.15.146:0/1043537959 conn(0x7efe740d0240 msgr2=0x7efe740d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 -- 172.21.15.146:0/1043537959 shutdown_connections 2024-04-25T15:21:12.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.085+0000 7efe7a6f9640 1 -- 172.21.15.146:0/1043537959 wait complete. 2024-04-25T15:21:12.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.086+0000 7efe7a6f9640 1 Processor -- start 2024-04-25T15:21:12.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.087+0000 7efe7a6f9640 1 -- start start 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe7a6f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 0x7efe740b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe7a6f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe740b6610 0x7efe740ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe7a6f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe7415ffa0 0x7efe7414c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe7a6f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efe741603b0 con 0x7efe740b5780 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe7a6f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efe740cfd60 con 0x7efe740b6610 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe7a6f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efe740d1da0 con 0x7efe7415ffa0 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 0x7efe740b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 0x7efe740b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38100/0 (socket says 172.21.15.146:38100) 2024-04-25T15:21:12.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 -- 172.21.15.146:0/331717216 learned_addr learned my addr 172.21.15.146:0/331717216 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe78c96640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe7415ffa0 0x7efe7414c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe737fe640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe740b6610 0x7efe740ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 -- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe7415ffa0 msgr2=0x7efe7414c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe7415ffa0 0x7efe7414c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 -- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe740b6610 msgr2=0x7efe740ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe740b6610 0x7efe740ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.088+0000 7efe73fff640 1 -- 172.21.15.146:0/331717216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7414a130 con 0x7efe740b5780 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.089+0000 7efe73fff640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 0x7efe740b0290 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efe6400e870 tx=0x7efe6400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.089+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe64017070 con 0x7efe740b5780 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.089+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6400c940 con 0x7efe740b5780 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.089+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe740aec10 con 0x7efe740b5780 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.089+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6400cd30 con 0x7efe740b5780 2024-04-25T15:21:12.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.089+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe7414a910 con 0x7efe740b5780 2024-04-25T15:21:12.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.090+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe7414b310 con 0x7efe740b5780 2024-04-25T15:21:12.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.091+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efe6401a070 con 0x7efe740b5780 2024-04-25T15:21:12.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.091+0000 7efe70ff9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efe600744b0 0x7efe600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.091+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7efe640951e0 con 0x7efe740b5780 2024-04-25T15:21:12.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.091+0000 7efe737fe640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efe600744b0 0x7efe600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.091+0000 7efe737fe640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efe600744b0 0x7efe600769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efe68034000 tx=0x7efe6802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.096+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe6405f190 con 0x7efe740b5780 2024-04-25T15:21:12.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.268+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe7406a0e0 con 0x7efe600744b0 2024-04-25T15:21:12.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.274+0000 7efe70ff9640 1 -- 172.21.15.146:0/331717216 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246623 (secure 0 0 0) 0x7efe7406a0e0 con 0x7efe600744b0 2024-04-25T15:21:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:12.279 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efe600744b0 msgr2=0x7efe600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efe600744b0 0x7efe600769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efe68034000 tx=0x7efe6802e000 comp rx=0 tx=0).stop 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 msgr2=0x7efe740b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 0x7efe740b0290 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7efe6400e870 tx=0x7efe6400ed30 comp rx=0 tx=0).stop 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 shutdown_connections 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efe600744b0 0x7efe600769c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe7415ffa0 0x7efe7414c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe740b6610 0x7efe740ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 --2- 172.21.15.146:0/331717216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe740b5780 0x7efe740b0290 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 >> 172.21.15.146:0/331717216 conn(0x7efe740d0240 msgr2=0x7efe740b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:12.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 shutdown_connections 2024-04-25T15:21:12.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.277+0000 7efe7a6f9640 1 -- 172.21.15.146:0/331717216 wait complete. 2024-04-25T15:21:12.289 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-25T15:21:12.005640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220644,"kb_used_data":4464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754076,"statfs":{"total":773094113280,"available":772868173824,"internally_reserved":0,"allocated":4571136,"data_stored":2632810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758904"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258465+0000","last_change":"2024-04-25T15:21:07.258465+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258465+0000","last_undegraded":"2024-04-25T15:21:07.258465+0000","last_fullsized":"2024-04-25T15:21:07.258465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259346+0000","last_change":"2024-04-25T15:21:07.259346+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259346+0000","last_undegraded":"2024-04-25T15:21:07.259346+0000","last_fullsized":"2024-04-25T15:21:07.259346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.265056+0000","last_change":"2024-04-25T15:21:07.265056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.265056+0000","last_undegraded":"2024-04-25T15:21:07.265056+0000","last_fullsized":"2024-04-25T15:21:07.265056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258425+0000","last_change":"2024-04-25T15:21:07.258425+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258425+0000","last_undegraded":"2024-04-25T15:21:07.258425+0000","last_fullsized":"2024-04-25T15:21:07.258425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260423+0000","last_change":"2024-04-25T15:21:07.260423+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260423+0000","last_undegraded":"2024-04-25T15:21:07.260423+0000","last_fullsized":"2024-04-25T15:21:07.260423+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256056+0000","last_change":"2024-04-25T15:21:07.256056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256056+0000","last_undegraded":"2024-04-25T15:21:07.256056+0000","last_fullsized":"2024-04-25T15:21:07.256056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253502+0000","last_change":"2024-04-25T15:21:07.253502+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253502+0000","last_undegraded":"2024-04-25T15:21:07.253502+0000","last_fullsized":"2024-04-25T15:21:07.253502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259316+0000","last_change":"2024-04-25T15:21:07.259316+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259316+0000","last_undegraded":"2024-04-25T15:21:07.259316+0000","last_fullsized":"2024-04-25T15:21:07.259316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.264325+0000","last_change":"2024-04-25T15:21:07.264325+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.264325+0000","last_undegraded":"2024-04-25T15:21:07.264325+0000","last_fullsized":"2024-04-25T15:21:07.264325+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252907+0000","last_change":"2024-04-25T15:21:07.252907+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252907+0000","last_undegraded":"2024-04-25T15:21:07.252907+0000","last_fullsized":"2024-04-25T15:21:07.252907+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256157+0000","last_change":"2024-04-25T15:21:07.256157+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256157+0000","last_undegraded":"2024-04-25T15:21:07.256157+0000","last_fullsized":"2024-04-25T15:21:07.256157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255957+0000","last_change":"2024-04-25T15:21:07.255957+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255957+0000","last_undegraded":"2024-04-25T15:21:07.255957+0000","last_fullsized":"2024-04-25T15:21:07.255957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255463+0000","last_change":"2024-04-25T15:21:07.255463+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255463+0000","last_undegraded":"2024-04-25T15:21:07.255463+0000","last_fullsized":"2024-04-25T15:21:07.255463+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257032+0000","last_change":"2024-04-25T15:21:07.257032+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257032+0000","last_undegraded":"2024-04-25T15:21:07.257032+0000","last_fullsized":"2024-04-25T15:21:07.257032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252787+0000","last_change":"2024-04-25T15:21:07.252787+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252787+0000","last_undegraded":"2024-04-25T15:21:07.252787+0000","last_fullsized":"2024-04-25T15:21:07.252787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258605+0000","last_change":"2024-04-25T15:21:07.258605+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258605+0000","last_undegraded":"2024-04-25T15:21:07.258605+0000","last_fullsized":"2024-04-25T15:21:07.258605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260477+0000","last_change":"2024-04-25T15:21:07.260477+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260477+0000","last_undegraded":"2024-04-25T15:21:07.260477+0000","last_fullsized":"2024-04-25T15:21:07.260477+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260775+0000","last_change":"2024-04-25T15:21:07.260775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260775+0000","last_undegraded":"2024-04-25T15:21:07.260775+0000","last_fullsized":"2024-04-25T15:21:07.260775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261610+0000","last_change":"2024-04-25T15:21:07.261610+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261610+0000","last_undegraded":"2024-04-25T15:21:07.261610+0000","last_fullsized":"2024-04-25T15:21:07.261610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260690+0000","last_change":"2024-04-25T15:21:07.260690+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260690+0000","last_undegraded":"2024-04-25T15:21:07.260690+0000","last_fullsized":"2024-04-25T15:21:07.260690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254693+0000","last_change":"2024-04-25T15:21:07.254693+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254693+0000","last_undegraded":"2024-04-25T15:21:07.254693+0000","last_fullsized":"2024-04-25T15:21:07.254693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255362+0000","last_change":"2024-04-25T15:21:07.255362+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255362+0000","last_undegraded":"2024-04-25T15:21:07.255362+0000","last_fullsized":"2024-04-25T15:21:07.255362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257566+0000","last_change":"2024-04-25T15:21:07.257566+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257566+0000","last_undegraded":"2024-04-25T15:21:07.257566+0000","last_fullsized":"2024-04-25T15:21:07.257566+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257299+0000","last_change":"2024-04-25T15:21:07.257299+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257299+0000","last_undegraded":"2024-04-25T15:21:07.257299+0000","last_fullsized":"2024-04-25T15:21:07.257299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261464+0000","last_change":"2024-04-25T15:21:07.261464+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261464+0000","last_undegraded":"2024-04-25T15:21:07.261464+0000","last_fullsized":"2024-04-25T15:21:07.261464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256914+0000","last_change":"2024-04-25T15:21:07.256914+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256914+0000","last_undegraded":"2024-04-25T15:21:07.256914+0000","last_fullsized":"2024-04-25T15:21:07.256914+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259855+0000","last_change":"2024-04-25T15:21:07.259855+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259855+0000","last_undegraded":"2024-04-25T15:21:07.259855+0000","last_fullsized":"2024-04-25T15:21:07.259855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252242+0000","last_change":"2024-04-25T15:21:07.252242+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252242+0000","last_undegraded":"2024-04-25T15:21:07.252242+0000","last_fullsized":"2024-04-25T15:21:07.252242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260409+0000","last_change":"2024-04-25T15:21:07.260409+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260409+0000","last_undegraded":"2024-04-25T15:21:07.260409+0000","last_fullsized":"2024-04-25T15:21:07.260409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259146+0000","last_change":"2024-04-25T15:21:07.259146+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259146+0000","last_undegraded":"2024-04-25T15:21:07.259146+0000","last_fullsized":"2024-04-25T15:21:07.259146+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255377+0000","last_change":"2024-04-25T15:21:07.255377+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255377+0000","last_undegraded":"2024-04-25T15:21:07.255377+0000","last_fullsized":"2024-04-25T15:21:07.255377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254666+0000","last_change":"2024-04-25T15:21:07.254666+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254666+0000","last_undegraded":"2024-04-25T15:21:07.254666+0000","last_fullsized":"2024-04-25T15:21:07.254666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257141+0000","last_change":"2024-04-25T15:21:07.257141+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257141+0000","last_undegraded":"2024-04-25T15:21:07.257141+0000","last_fullsized":"2024-04-25T15:21:07.257141+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277551+0000","last_change":"2024-04-25T15:21:07.273191+0000","last_active":"2024-04-25T15:21:07.277551+0000","last_peered":"2024-04-25T15:21:07.277551+0000","last_clean":"2024-04-25T15:21:07.277551+0000","last_became_active":"2024-04-25T15:21:07.272854+0000","last_became_peered":"2024-04-25T15:21:07.272854+0000","last_unstale":"2024-04-25T15:21:07.277551+0000","last_undegraded":"2024-04-25T15:21:07.277551+0000","last_fullsized":"2024-04-25T15:21:07.277551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:00.687305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257143+0000","last_change":"2024-04-25T15:21:07.257143+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257143+0000","last_undegraded":"2024-04-25T15:21:07.257143+0000","last_fullsized":"2024-04-25T15:21:07.257143+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270022+0000","last_change":"2024-04-25T15:21:07.268440+0000","last_active":"2024-04-25T15:21:07.270022+0000","last_peered":"2024-04-25T15:21:07.270022+0000","last_clean":"2024-04-25T15:21:07.270022+0000","last_became_active":"2024-04-25T15:21:07.268125+0000","last_became_peered":"2024-04-25T15:21:07.268125+0000","last_unstale":"2024-04-25T15:21:07.270022+0000","last_undegraded":"2024-04-25T15:21:07.270022+0000","last_fullsized":"2024-04-25T15:21:07.270022+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254113+0000","last_change":"2024-04-25T15:21:07.254113+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254113+0000","last_undegraded":"2024-04-25T15:21:07.254113+0000","last_fullsized":"2024-04-25T15:21:07.254113+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272789+0000","last_change":"2024-04-25T15:21:07.266826+0000","last_active":"2024-04-25T15:21:07.272789+0000","last_peered":"2024-04-25T15:21:07.272789+0000","last_clean":"2024-04-25T15:21:07.272789+0000","last_became_active":"2024-04-25T15:21:07.266540+0000","last_became_peered":"2024-04-25T15:21:07.266540+0000","last_unstale":"2024-04-25T15:21:07.272789+0000","last_undegraded":"2024-04-25T15:21:07.272789+0000","last_fullsized":"2024-04-25T15:21:07.272789+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258984+0000","last_change":"2024-04-25T15:21:07.258984+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258984+0000","last_undegraded":"2024-04-25T15:21:07.258984+0000","last_fullsized":"2024-04-25T15:21:07.258984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272412+0000","last_change":"2024-04-25T15:21:07.268165+0000","last_active":"2024-04-25T15:21:07.272412+0000","last_peered":"2024-04-25T15:21:07.272412+0000","last_clean":"2024-04-25T15:21:07.272412+0000","last_became_active":"2024-04-25T15:21:07.267821+0000","last_became_peered":"2024-04-25T15:21:07.267821+0000","last_unstale":"2024-04-25T15:21:07.272412+0000","last_undegraded":"2024-04-25T15:21:07.272412+0000","last_fullsized":"2024-04-25T15:21:07.272412+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255378+0000","last_change":"2024-04-25T15:21:07.255378+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255378+0000","last_undegraded":"2024-04-25T15:21:07.255378+0000","last_fullsized":"2024-04-25T15:21:07.255378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.260943+0000","last_change":"2024-04-25T15:21:07.258767+0000","last_active":"2024-04-25T15:21:07.260943+0000","last_peered":"2024-04-25T15:21:07.260943+0000","last_clean":"2024-04-25T15:21:07.260943+0000","last_became_active":"2024-04-25T15:21:07.258461+0000","last_became_peered":"2024-04-25T15:21:07.258461+0000","last_unstale":"2024-04-25T15:21:07.260943+0000","last_undegraded":"2024-04-25T15:21:07.260943+0000","last_fullsized":"2024-04-25T15:21:07.260943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252100+0000","last_change":"2024-04-25T15:21:07.252100+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252100+0000","last_undegraded":"2024-04-25T15:21:07.252100+0000","last_fullsized":"2024-04-25T15:21:07.252100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280453+0000","last_change":"2024-04-25T15:21:07.273310+0000","last_active":"2024-04-25T15:21:07.280453+0000","last_peered":"2024-04-25T15:21:07.280453+0000","last_clean":"2024-04-25T15:21:07.280453+0000","last_became_active":"2024-04-25T15:21:07.272977+0000","last_became_peered":"2024-04-25T15:21:07.272977+0000","last_unstale":"2024-04-25T15:21:07.280453+0000","last_undegraded":"2024-04-25T15:21:07.280453+0000","last_fullsized":"2024-04-25T15:21:07.280453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253544+0000","last_change":"2024-04-25T15:21:07.253544+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253544+0000","last_undegraded":"2024-04-25T15:21:07.253544+0000","last_fullsized":"2024-04-25T15:21:07.253544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.271664+0000","last_change":"2024-04-25T15:21:07.263740+0000","last_active":"2024-04-25T15:21:07.271664+0000","last_peered":"2024-04-25T15:21:07.271664+0000","last_clean":"2024-04-25T15:21:07.271664+0000","last_became_active":"2024-04-25T15:21:07.262783+0000","last_became_peered":"2024-04-25T15:21:07.262783+0000","last_unstale":"2024-04-25T15:21:07.271664+0000","last_undegraded":"2024-04-25T15:21:07.271664+0000","last_fullsized":"2024-04-25T15:21:07.271664+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258866+0000","last_change":"2024-04-25T15:21:07.258866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258866+0000","last_undegraded":"2024-04-25T15:21:07.258866+0000","last_fullsized":"2024-04-25T15:21:07.258866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272273+0000","last_change":"2024-04-25T15:21:07.267048+0000","last_active":"2024-04-25T15:21:07.272273+0000","last_peered":"2024-04-25T15:21:07.272273+0000","last_clean":"2024-04-25T15:21:07.272273+0000","last_became_active":"2024-04-25T15:21:07.265790+0000","last_became_peered":"2024-04-25T15:21:07.265790+0000","last_unstale":"2024-04-25T15:21:07.272273+0000","last_undegraded":"2024-04-25T15:21:07.272273+0000","last_fullsized":"2024-04-25T15:21:07.272273+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257928+0000","last_change":"2024-04-25T15:21:07.257928+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257928+0000","last_undegraded":"2024-04-25T15:21:07.257928+0000","last_fullsized":"2024-04-25T15:21:07.257928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279451+0000","last_change":"2024-04-25T15:21:07.273306+0000","last_active":"2024-04-25T15:21:07.279451+0000","last_peered":"2024-04-25T15:21:07.279451+0000","last_clean":"2024-04-25T15:21:07.279451+0000","last_became_active":"2024-04-25T15:21:07.272979+0000","last_became_peered":"2024-04-25T15:21:07.272979+0000","last_unstale":"2024-04-25T15:21:07.279451+0000","last_undegraded":"2024-04-25T15:21:07.279451+0000","last_fullsized":"2024-04-25T15:21:07.279451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:09:56.214534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.262262+0000","last_change":"2024-04-25T15:21:07.262262+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.262262+0000","last_undegraded":"2024-04-25T15:21:07.262262+0000","last_fullsized":"2024-04-25T15:21:07.262262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270661+0000","last_change":"2024-04-25T15:21:07.268388+0000","last_active":"2024-04-25T15:21:07.270661+0000","last_peered":"2024-04-25T15:21:07.270661+0000","last_clean":"2024-04-25T15:21:07.270661+0000","last_became_active":"2024-04-25T15:21:07.267977+0000","last_became_peered":"2024-04-25T15:21:07.267977+0000","last_unstale":"2024-04-25T15:21:07.270661+0000","last_undegraded":"2024-04-25T15:21:07.270661+0000","last_fullsized":"2024-04-25T15:21:07.270661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254905+0000","last_change":"2024-04-25T15:21:07.254905+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254905+0000","last_undegraded":"2024-04-25T15:21:07.254905+0000","last_fullsized":"2024-04-25T15:21:07.254905+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270271+0000","last_change":"2024-04-25T15:21:07.263141+0000","last_active":"2024-04-25T15:21:07.270271+0000","last_peered":"2024-04-25T15:21:07.270271+0000","last_clean":"2024-04-25T15:21:07.270271+0000","last_became_active":"2024-04-25T15:21:07.262765+0000","last_became_peered":"2024-04-25T15:21:07.262765+0000","last_unstale":"2024-04-25T15:21:07.270271+0000","last_undegraded":"2024-04-25T15:21:07.270271+0000","last_fullsized":"2024-04-25T15:21:07.270271+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252968+0000","last_change":"2024-04-25T15:21:07.252968+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252968+0000","last_undegraded":"2024-04-25T15:21:07.252968+0000","last_fullsized":"2024-04-25T15:21:07.252968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272311+0000","last_change":"2024-04-25T15:21:07.268503+0000","last_active":"2024-04-25T15:21:07.272311+0000","last_peered":"2024-04-25T15:21:07.272311+0000","last_clean":"2024-04-25T15:21:07.272311+0000","last_became_active":"2024-04-25T15:21:07.268088+0000","last_became_peered":"2024-04-25T15:21:07.268088+0000","last_unstale":"2024-04-25T15:21:07.272311+0000","last_undegraded":"2024-04-25T15:21:07.272311+0000","last_fullsized":"2024-04-25T15:21:07.272311+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:36.404924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254326+0000","last_change":"2024-04-25T15:21:07.254326+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254326+0000","last_undegraded":"2024-04-25T15:21:07.254326+0000","last_fullsized":"2024-04-25T15:21:07.254326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272165+0000","last_change":"2024-04-25T15:21:07.267053+0000","last_active":"2024-04-25T15:21:07.272165+0000","last_peered":"2024-04-25T15:21:07.272165+0000","last_clean":"2024-04-25T15:21:07.272165+0000","last_became_active":"2024-04-25T15:21:07.266690+0000","last_became_peered":"2024-04-25T15:21:07.266690+0000","last_unstale":"2024-04-25T15:21:07.272165+0000","last_undegraded":"2024-04-25T15:21:07.272165+0000","last_fullsized":"2024-04-25T15:21:07.272165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:47.899553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257171+0000","last_change":"2024-04-25T15:21:07.257171+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257171+0000","last_undegraded":"2024-04-25T15:21:07.257171+0000","last_fullsized":"2024-04-25T15:21:07.257171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272200+0000","last_change":"2024-04-25T15:21:07.262699+0000","last_active":"2024-04-25T15:21:07.272200+0000","last_peered":"2024-04-25T15:21:07.272200+0000","last_clean":"2024-04-25T15:21:07.272200+0000","last_became_active":"2024-04-25T15:21:07.262384+0000","last_became_peered":"2024-04-25T15:21:07.262384+0000","last_unstale":"2024-04-25T15:21:07.272200+0000","last_undegraded":"2024-04-25T15:21:07.272200+0000","last_fullsized":"2024-04-25T15:21:07.272200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256033+0000","last_change":"2024-04-25T15:21:07.256033+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256033+0000","last_undegraded":"2024-04-25T15:21:07.256033+0000","last_fullsized":"2024-04-25T15:21:07.256033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274736+0000","last_change":"2024-04-25T15:21:07.270561+0000","last_active":"2024-04-25T15:21:07.274736+0000","last_peered":"2024-04-25T15:21:07.274736+0000","last_clean":"2024-04-25T15:21:07.274736+0000","last_became_active":"2024-04-25T15:21:07.270106+0000","last_became_peered":"2024-04-25T15:21:07.270106+0000","last_unstale":"2024-04-25T15:21:07.274736+0000","last_undegraded":"2024-04-25T15:21:07.274736+0000","last_fullsized":"2024-04-25T15:21:07.274736+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:14:54.724610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256795+0000","last_change":"2024-04-25T15:21:07.256795+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256795+0000","last_undegraded":"2024-04-25T15:21:07.256795+0000","last_fullsized":"2024-04-25T15:21:07.256795+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280489+0000","last_change":"2024-04-25T15:21:07.273386+0000","last_active":"2024-04-25T15:21:07.280489+0000","last_peered":"2024-04-25T15:21:07.280489+0000","last_clean":"2024-04-25T15:21:07.280489+0000","last_became_active":"2024-04-25T15:21:07.272939+0000","last_became_peered":"2024-04-25T15:21:07.272939+0000","last_unstale":"2024-04-25T15:21:07.280489+0000","last_undegraded":"2024-04-25T15:21:07.280489+0000","last_fullsized":"2024-04-25T15:21:07.280489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259420+0000","last_change":"2024-04-25T15:21:07.259420+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259420+0000","last_undegraded":"2024-04-25T15:21:07.259420+0000","last_fullsized":"2024-04-25T15:21:07.259420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270390+0000","last_change":"2024-04-25T15:21:07.266432+0000","last_active":"2024-04-25T15:21:07.270390+0000","last_peered":"2024-04-25T15:21:07.270390+0000","last_clean":"2024-04-25T15:21:07.270390+0000","last_became_active":"2024-04-25T15:21:07.266089+0000","last_became_peered":"2024-04-25T15:21:07.266089+0000","last_unstale":"2024-04-25T15:21:07.270390+0000","last_undegraded":"2024-04-25T15:21:07.270390+0000","last_fullsized":"2024-04-25T15:21:07.270390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254229+0000","last_change":"2024-04-25T15:21:07.254229+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254229+0000","last_undegraded":"2024-04-25T15:21:07.254229+0000","last_fullsized":"2024-04-25T15:21:07.254229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269440+0000","last_change":"2024-04-25T15:21:07.262181+0000","last_active":"2024-04-25T15:21:07.269440+0000","last_peered":"2024-04-25T15:21:07.269440+0000","last_clean":"2024-04-25T15:21:07.269440+0000","last_became_active":"2024-04-25T15:21:07.261828+0000","last_became_peered":"2024-04-25T15:21:07.261828+0000","last_unstale":"2024-04-25T15:21:07.269440+0000","last_undegraded":"2024-04-25T15:21:07.269440+0000","last_fullsized":"2024-04-25T15:21:07.269440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256308+0000","last_change":"2024-04-25T15:21:07.256308+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256308+0000","last_undegraded":"2024-04-25T15:21:07.256308+0000","last_fullsized":"2024-04-25T15:21:07.256308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276901+0000","last_change":"2024-04-25T15:21:07.270193+0000","last_active":"2024-04-25T15:21:07.276901+0000","last_peered":"2024-04-25T15:21:07.276901+0000","last_clean":"2024-04-25T15:21:07.276901+0000","last_became_active":"2024-04-25T15:21:07.269650+0000","last_became_peered":"2024-04-25T15:21:07.269650+0000","last_unstale":"2024-04-25T15:21:07.276901+0000","last_undegraded":"2024-04-25T15:21:07.276901+0000","last_fullsized":"2024-04-25T15:21:07.276901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:17.140514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258937+0000","last_change":"2024-04-25T15:21:07.258937+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258937+0000","last_undegraded":"2024-04-25T15:21:07.258937+0000","last_fullsized":"2024-04-25T15:21:07.258937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.273489+0000","last_change":"2024-04-25T15:21:07.270403+0000","last_active":"2024-04-25T15:21:07.273489+0000","last_peered":"2024-04-25T15:21:07.273489+0000","last_clean":"2024-04-25T15:21:07.273489+0000","last_became_active":"2024-04-25T15:21:07.270082+0000","last_became_peered":"2024-04-25T15:21:07.270082+0000","last_unstale":"2024-04-25T15:21:07.273489+0000","last_undegraded":"2024-04-25T15:21:07.273489+0000","last_fullsized":"2024-04-25T15:21:07.273489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256563+0000","last_change":"2024-04-25T15:21:07.256563+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256563+0000","last_undegraded":"2024-04-25T15:21:07.256563+0000","last_fullsized":"2024-04-25T15:21:07.256563+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276978+0000","last_change":"2024-04-25T15:21:07.270988+0000","last_active":"2024-04-25T15:21:07.276978+0000","last_peered":"2024-04-25T15:21:07.276978+0000","last_clean":"2024-04-25T15:21:07.276978+0000","last_became_active":"2024-04-25T15:21:07.270686+0000","last_became_peered":"2024-04-25T15:21:07.270686+0000","last_unstale":"2024-04-25T15:21:07.276978+0000","last_undegraded":"2024-04-25T15:21:07.276978+0000","last_fullsized":"2024-04-25T15:21:07.276978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:14.556209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253688+0000","last_change":"2024-04-25T15:21:07.253688+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253688+0000","last_undegraded":"2024-04-25T15:21:07.253688+0000","last_fullsized":"2024-04-25T15:21:07.253688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276948+0000","last_change":"2024-04-25T15:21:07.270720+0000","last_active":"2024-04-25T15:21:07.276948+0000","last_peered":"2024-04-25T15:21:07.276948+0000","last_clean":"2024-04-25T15:21:07.276948+0000","last_became_active":"2024-04-25T15:21:07.270418+0000","last_became_peered":"2024-04-25T15:21:07.270418+0000","last_unstale":"2024-04-25T15:21:07.276948+0000","last_undegraded":"2024-04-25T15:21:07.276948+0000","last_fullsized":"2024-04-25T15:21:07.276948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:23.664366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254153+0000","last_change":"2024-04-25T15:21:07.254153+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254153+0000","last_undegraded":"2024-04-25T15:21:07.254153+0000","last_fullsized":"2024-04-25T15:21:07.254153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274957+0000","last_change":"2024-04-25T15:21:07.270169+0000","last_active":"2024-04-25T15:21:07.274957+0000","last_peered":"2024-04-25T15:21:07.274957+0000","last_clean":"2024-04-25T15:21:07.274957+0000","last_became_active":"2024-04-25T15:21:07.269753+0000","last_became_peered":"2024-04-25T15:21:07.269753+0000","last_unstale":"2024-04-25T15:21:07.274957+0000","last_undegraded":"2024-04-25T15:21:07.274957+0000","last_fullsized":"2024-04-25T15:21:07.274957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:37.213221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257535+0000","last_change":"2024-04-25T15:21:07.257535+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257535+0000","last_undegraded":"2024-04-25T15:21:07.257535+0000","last_fullsized":"2024-04-25T15:21:07.257535+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272859+0000","last_change":"2024-04-25T15:21:07.269932+0000","last_active":"2024-04-25T15:21:07.272859+0000","last_peered":"2024-04-25T15:21:07.272859+0000","last_clean":"2024-04-25T15:21:07.272859+0000","last_became_active":"2024-04-25T15:21:07.269598+0000","last_became_peered":"2024-04-25T15:21:07.269598+0000","last_unstale":"2024-04-25T15:21:07.272859+0000","last_undegraded":"2024-04-25T15:21:07.272859+0000","last_fullsized":"2024-04-25T15:21:07.272859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:39.459934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256124+0000","last_change":"2024-04-25T15:21:07.256124+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256124+0000","last_undegraded":"2024-04-25T15:21:07.256124+0000","last_fullsized":"2024-04-25T15:21:07.256124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270365+0000","last_change":"2024-04-25T15:21:07.263451+0000","last_active":"2024-04-25T15:21:07.270365+0000","last_peered":"2024-04-25T15:21:07.270365+0000","last_clean":"2024-04-25T15:21:07.270365+0000","last_became_active":"2024-04-25T15:21:07.263111+0000","last_became_peered":"2024-04-25T15:21:07.263111+0000","last_unstale":"2024-04-25T15:21:07.270365+0000","last_undegraded":"2024-04-25T15:21:07.270365+0000","last_fullsized":"2024-04-25T15:21:07.270365+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261818+0000","last_change":"2024-04-25T15:21:07.261818+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261818+0000","last_undegraded":"2024-04-25T15:21:07.261818+0000","last_fullsized":"2024-04-25T15:21:07.261818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277011+0000","last_change":"2024-04-25T15:21:07.271421+0000","last_active":"2024-04-25T15:21:07.277011+0000","last_peered":"2024-04-25T15:21:07.277011+0000","last_clean":"2024-04-25T15:21:07.277011+0000","last_became_active":"2024-04-25T15:21:07.271136+0000","last_became_peered":"2024-04-25T15:21:07.271136+0000","last_unstale":"2024-04-25T15:21:07.277011+0000","last_undegraded":"2024-04-25T15:21:07.277011+0000","last_fullsized":"2024-04-25T15:21:07.277011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:34.931594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258775+0000","last_change":"2024-04-25T15:21:07.258775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258775+0000","last_undegraded":"2024-04-25T15:21:07.258775+0000","last_fullsized":"2024-04-25T15:21:07.258775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269969+0000","last_change":"2024-04-25T15:21:07.267201+0000","last_active":"2024-04-25T15:21:07.269969+0000","last_peered":"2024-04-25T15:21:07.269969+0000","last_clean":"2024-04-25T15:21:07.269969+0000","last_became_active":"2024-04-25T15:21:07.266901+0000","last_became_peered":"2024-04-25T15:21:07.266901+0000","last_unstale":"2024-04-25T15:21:07.269969+0000","last_undegraded":"2024-04-25T15:21:07.269969+0000","last_fullsized":"2024-04-25T15:21:07.269969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253682+0000","last_change":"2024-04-25T15:21:07.253682+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253682+0000","last_undegraded":"2024-04-25T15:21:07.253682+0000","last_fullsized":"2024-04-25T15:21:07.253682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270067+0000","last_change":"2024-04-25T15:21:07.265869+0000","last_active":"2024-04-25T15:21:07.270067+0000","last_peered":"2024-04-25T15:21:07.270067+0000","last_clean":"2024-04-25T15:21:07.270067+0000","last_became_active":"2024-04-25T15:21:07.265566+0000","last_became_peered":"2024-04-25T15:21:07.265566+0000","last_unstale":"2024-04-25T15:21:07.270067+0000","last_undegraded":"2024-04-25T15:21:07.270067+0000","last_fullsized":"2024-04-25T15:21:07.270067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256908+0000","last_change":"2024-04-25T15:21:07.256908+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256908+0000","last_undegraded":"2024-04-25T15:21:07.256908+0000","last_fullsized":"2024-04-25T15:21:07.256908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272141+0000","last_change":"2024-04-25T15:21:07.267132+0000","last_active":"2024-04-25T15:21:07.272141+0000","last_peered":"2024-04-25T15:21:07.272141+0000","last_clean":"2024-04-25T15:21:07.272141+0000","last_became_active":"2024-04-25T15:21:07.266783+0000","last_became_peered":"2024-04-25T15:21:07.266783+0000","last_unstale":"2024-04-25T15:21:07.272141+0000","last_undegraded":"2024-04-25T15:21:07.272141+0000","last_fullsized":"2024-04-25T15:21:07.272141+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:10.461067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255866+0000","last_change":"2024-04-25T15:21:07.255866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255866+0000","last_undegraded":"2024-04-25T15:21:07.255866+0000","last_fullsized":"2024-04-25T15:21:07.255866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272473+0000","last_change":"2024-04-25T15:21:07.268347+0000","last_active":"2024-04-25T15:21:07.272473+0000","last_peered":"2024-04-25T15:21:07.272473+0000","last_clean":"2024-04-25T15:21:07.272473+0000","last_became_active":"2024-04-25T15:21:07.268032+0000","last_became_peered":"2024-04-25T15:21:07.268032+0000","last_unstale":"2024-04-25T15:21:07.272473+0000","last_undegraded":"2024-04-25T15:21:07.272473+0000","last_fullsized":"2024-04-25T15:21:07.272473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260480+0000","last_change":"2024-04-25T15:21:07.260480+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260480+0000","last_undegraded":"2024-04-25T15:21:07.260480+0000","last_fullsized":"2024-04-25T15:21:07.260480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279438+0000","last_change":"2024-04-25T15:21:07.272594+0000","last_active":"2024-04-25T15:21:07.279438+0000","last_peered":"2024-04-25T15:21:07.279438+0000","last_clean":"2024-04-25T15:21:07.279438+0000","last_became_active":"2024-04-25T15:21:07.272274+0000","last_became_peered":"2024-04-25T15:21:07.272274+0000","last_unstale":"2024-04-25T15:21:07.279438+0000","last_undegraded":"2024-04-25T15:21:07.279438+0000","last_fullsized":"2024-04-25T15:21:07.279438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:09.551367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254935+0000","last_change":"2024-04-25T15:21:07.254935+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254935+0000","last_undegraded":"2024-04-25T15:21:07.254935+0000","last_fullsized":"2024-04-25T15:21:07.254935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274974+0000","last_change":"2024-04-25T15:21:07.271612+0000","last_active":"2024-04-25T15:21:07.274974+0000","last_peered":"2024-04-25T15:21:07.274974+0000","last_clean":"2024-04-25T15:21:07.274974+0000","last_became_active":"2024-04-25T15:21:07.271320+0000","last_became_peered":"2024-04-25T15:21:07.271320+0000","last_unstale":"2024-04-25T15:21:07.274974+0000","last_undegraded":"2024-04-25T15:21:07.274974+0000","last_fullsized":"2024-04-25T15:21:07.274974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:42.520351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":141,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.259258+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:21:07.259258+0000","last_peered":"2024-04-25T15:21:07.259258+0000","last_clean":"2024-04-25T15:21:07.259258+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:21:07.259258+0000","last_undegraded":"2024-04-25T15:21:07.259258+0000","last_fullsized":"2024-04-25T15:21:07.259258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1019904,"data_stored":773544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:21:12.294 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:21:12.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.436+0000 7f0c0baaf640 1 Processor -- start 2024-04-25T15:21:12.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.436+0000 7f0c0baaf640 1 -- start start 2024-04-25T15:21:12.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0baaf640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0c04160e00 0x7f0c0415c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0baaf640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0406cde0 0x7f0c0406d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0baaf640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c040b10b0 con 0x7f0c040ccfb0 2024-04-25T15:21:12.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0baaf640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0c0415c9d0 con 0x7f0c04160e00 2024-04-25T15:21:12.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0baaf640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0c041600d0 con 0x7f0c0406cde0 2024-04-25T15:21:12.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0984b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0c04160e00 0x7f0c0415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0904a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0406cde0 0x7f0c0406d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0a04c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f0c040ccfb0 0x7f0c040cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:58826/0 (socket says 172.21.15.146:58826) 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.438+0000 7f0c0a04c640 1 -- 172.21.15.146:0/3905893195 learned_addr learned my addr 172.21.15.146:0/3905893195 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.439+0000 7f0c0904a640 1 -- 172.21.15.146:0/3905893195 >> v2:172.21.15.177:3300/0 conn(0x7f0c04160e00 msgr2=0x7f0c0415c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.439+0000 7f0c0904a640 1 --2- 172.21.15.146:0/3905893195 >> v2:172.21.15.177:3300/0 conn(0x7f0c04160e00 0x7f0c0415c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.439+0000 7f0c0904a640 1 -- 172.21.15.146:0/3905893195 >> v1:172.21.15.177:6789/0 conn(0x7f0c040ccfb0 legacy=0x7f0c040cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.439+0000 7f0c0904a640 1 -- 172.21.15.146:0/3905893195 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0406d730 con 0x7f0c0406cde0 2024-04-25T15:21:12.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.439+0000 7f0c0984b640 1 --2- 172.21.15.146:0/3905893195 >> v2:172.21.15.177:3300/0 conn(0x7f0c04160e00 0x7f0c0415c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:12.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.439+0000 7f0c0904a640 1 --2- 172.21.15.146:0/3905893195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0406cde0 0x7f0c0406d1f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80097a0 tx=0x7f0bf802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5aaaf061aea1564 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c08849640 1 -- 172.21.15.146:0/3905893195 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8037070 con 0x7f0c0406cde0 2024-04-25T15:21:12.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c08849640 1 -- 172.21.15.146:0/3905893195 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf8032440 con 0x7f0c0406cde0 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c08849640 1 -- 172.21.15.146:0/3905893195 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8032810 con 0x7f0c0406cde0 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 -- 172.21.15.146:0/3905893195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0406cde0 msgr2=0x7f0c0406d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/3905893195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0406cde0 0x7f0c0406d1f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80097a0 tx=0x7f0bf802a180 comp rx=0 tx=0).stop 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 -- 172.21.15.146:0/3905893195 shutdown_connections 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/3905893195 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0406cde0 0x7f0c0406d1f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/3905893195 >> v2:172.21.15.177:3300/0 conn(0x7f0c04160e00 0x7f0c0415c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 -- 172.21.15.146:0/3905893195 >> 172.21.15.146:0/3905893195 conn(0x7f0c040ce0c0 msgr2=0x7f0c040ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 -- 172.21.15.146:0/3905893195 shutdown_connections 2024-04-25T15:21:12.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.440+0000 7f0c0baaf640 1 -- 172.21.15.146:0/3905893195 wait complete. 2024-04-25T15:21:12.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.442+0000 7f0c0baaf640 1 Processor -- start 2024-04-25T15:21:12.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.442+0000 7f0c0baaf640 1 -- start start 2024-04-25T15:21:12.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.443+0000 7f0c0baaf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0406cde0 0x7f0c04152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.443+0000 7f0c0baaf640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c040ccfb0 0x7f0c041f8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.443+0000 7f0c0baaf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 0x7f0c041f8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.443+0000 7f0c0baaf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0c04160600 con 0x7f0c040b6d50 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.443+0000 7f0c0baaf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c04160320 con 0x7f0c0406cde0 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.443+0000 7f0c0baaf640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0c041608e0 con 0x7f0c040ccfb0 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 0x7f0c041f8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 0x7f0c041f8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38102/0 (socket says 172.21.15.146:38102) 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 -- 172.21.15.146:0/4247030287 learned_addr learned my addr 172.21.15.146:0/4247030287 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0984b640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0406cde0 0x7f0c04152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0904a640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c040ccfb0 0x7f0c041f8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 -- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c040ccfb0 msgr2=0x7f0c041f8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c040ccfb0 0x7f0c041f8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 -- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0406cde0 msgr2=0x7f0c04152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0406cde0 0x7f0c04152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 -- 172.21.15.146:0/4247030287 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c041525a0 con 0x7f0c040b6d50 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0a04c640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 0x7f0c041f8cd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400cad0 tx=0x7f0bf400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4017070 con 0x7f0c040b6d50 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf400ac30 con 0x7f0c040b6d50 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4012400 con 0x7f0c040b6d50 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c040b56b0 con 0x7f0c040b6d50 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.444+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c040b5c70 con 0x7f0c040b6d50 2024-04-25T15:21:12.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.445+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c040b5420 con 0x7f0c040b6d50 2024-04-25T15:21:12.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.446+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0bf401a070 con 0x7f0c040b6d50 2024-04-25T15:21:12.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.446+0000 7f0c027fc640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bf00744b0 0x7f0bf00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.447+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0bf4095000 con 0x7f0c040b6d50 2024-04-25T15:21:12.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.447+0000 7f0c0984b640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bf00744b0 0x7f0bf00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.447+0000 7f0c0984b640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bf00744b0 0x7f0bf00769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b820 tx=0x7f0bfc00f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.451+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bf405efb0 con 0x7f0c040b6d50 2024-04-25T15:21:12.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.622+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c0406a0e0 con 0x7f0bf00744b0 2024-04-25T15:21:12.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.627+0000 7f0c027fc640 1 -- 172.21.15.146:0/4247030287 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246623 (secure 0 0 0) 0x7f0c0406a0e0 con 0x7f0bf00744b0 2024-04-25T15:21:12.631 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:12.632 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:21:12.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bf00744b0 msgr2=0x7f0bf00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bf00744b0 0x7f0bf00769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b820 tx=0x7f0bfc00f040 comp rx=0 tx=0).stop 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 msgr2=0x7f0c041f8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 0x7f0c041f8cd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400cad0 tx=0x7f0bf400cf90 comp rx=0 tx=0).stop 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 shutdown_connections 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0bf00744b0 0x7f0bf00769c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c040b6d50 0x7f0c041f8cd0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c040ccfb0 0x7f0c041f8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 --2- 172.21.15.146:0/4247030287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0406cde0 0x7f0c04152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 >> 172.21.15.146:0/4247030287 conn(0x7f0c040ce0c0 msgr2=0x7f0c04159280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 shutdown_connections 2024-04-25T15:21:12.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.631+0000 7f0c0baaf640 1 -- 172.21.15.146:0/4247030287 wait complete. 2024-04-25T15:21:12.643 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-25T15:21:12.005640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220644,"kb_used_data":4464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754076,"statfs":{"total":773094113280,"available":772868173824,"internally_reserved":0,"allocated":4571136,"data_stored":2632810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758904"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258465+0000","last_change":"2024-04-25T15:21:07.258465+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258465+0000","last_undegraded":"2024-04-25T15:21:07.258465+0000","last_fullsized":"2024-04-25T15:21:07.258465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259346+0000","last_change":"2024-04-25T15:21:07.259346+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259346+0000","last_undegraded":"2024-04-25T15:21:07.259346+0000","last_fullsized":"2024-04-25T15:21:07.259346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.265056+0000","last_change":"2024-04-25T15:21:07.265056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.265056+0000","last_undegraded":"2024-04-25T15:21:07.265056+0000","last_fullsized":"2024-04-25T15:21:07.265056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258425+0000","last_change":"2024-04-25T15:21:07.258425+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258425+0000","last_undegraded":"2024-04-25T15:21:07.258425+0000","last_fullsized":"2024-04-25T15:21:07.258425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260423+0000","last_change":"2024-04-25T15:21:07.260423+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260423+0000","last_undegraded":"2024-04-25T15:21:07.260423+0000","last_fullsized":"2024-04-25T15:21:07.260423+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256056+0000","last_change":"2024-04-25T15:21:07.256056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256056+0000","last_undegraded":"2024-04-25T15:21:07.256056+0000","last_fullsized":"2024-04-25T15:21:07.256056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253502+0000","last_change":"2024-04-25T15:21:07.253502+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253502+0000","last_undegraded":"2024-04-25T15:21:07.253502+0000","last_fullsized":"2024-04-25T15:21:07.253502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259316+0000","last_change":"2024-04-25T15:21:07.259316+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259316+0000","last_undegraded":"2024-04-25T15:21:07.259316+0000","last_fullsized":"2024-04-25T15:21:07.259316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.264325+0000","last_change":"2024-04-25T15:21:07.264325+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.264325+0000","last_undegraded":"2024-04-25T15:21:07.264325+0000","last_fullsized":"2024-04-25T15:21:07.264325+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252907+0000","last_change":"2024-04-25T15:21:07.252907+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252907+0000","last_undegraded":"2024-04-25T15:21:07.252907+0000","last_fullsized":"2024-04-25T15:21:07.252907+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256157+0000","last_change":"2024-04-25T15:21:07.256157+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256157+0000","last_undegraded":"2024-04-25T15:21:07.256157+0000","last_fullsized":"2024-04-25T15:21:07.256157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255957+0000","last_change":"2024-04-25T15:21:07.255957+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255957+0000","last_undegraded":"2024-04-25T15:21:07.255957+0000","last_fullsized":"2024-04-25T15:21:07.255957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255463+0000","last_change":"2024-04-25T15:21:07.255463+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255463+0000","last_undegraded":"2024-04-25T15:21:07.255463+0000","last_fullsized":"2024-04-25T15:21:07.255463+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257032+0000","last_change":"2024-04-25T15:21:07.257032+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257032+0000","last_undegraded":"2024-04-25T15:21:07.257032+0000","last_fullsized":"2024-04-25T15:21:07.257032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252787+0000","last_change":"2024-04-25T15:21:07.252787+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252787+0000","last_undegraded":"2024-04-25T15:21:07.252787+0000","last_fullsized":"2024-04-25T15:21:07.252787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258605+0000","last_change":"2024-04-25T15:21:07.258605+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258605+0000","last_undegraded":"2024-04-25T15:21:07.258605+0000","last_fullsized":"2024-04-25T15:21:07.258605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260477+0000","last_change":"2024-04-25T15:21:07.260477+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260477+0000","last_undegraded":"2024-04-25T15:21:07.260477+0000","last_fullsized":"2024-04-25T15:21:07.260477+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260775+0000","last_change":"2024-04-25T15:21:07.260775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260775+0000","last_undegraded":"2024-04-25T15:21:07.260775+0000","last_fullsized":"2024-04-25T15:21:07.260775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261610+0000","last_change":"2024-04-25T15:21:07.261610+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261610+0000","last_undegraded":"2024-04-25T15:21:07.261610+0000","last_fullsized":"2024-04-25T15:21:07.261610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260690+0000","last_change":"2024-04-25T15:21:07.260690+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260690+0000","last_undegraded":"2024-04-25T15:21:07.260690+0000","last_fullsized":"2024-04-25T15:21:07.260690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254693+0000","last_change":"2024-04-25T15:21:07.254693+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254693+0000","last_undegraded":"2024-04-25T15:21:07.254693+0000","last_fullsized":"2024-04-25T15:21:07.254693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255362+0000","last_change":"2024-04-25T15:21:07.255362+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255362+0000","last_undegraded":"2024-04-25T15:21:07.255362+0000","last_fullsized":"2024-04-25T15:21:07.255362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257566+0000","last_change":"2024-04-25T15:21:07.257566+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257566+0000","last_undegraded":"2024-04-25T15:21:07.257566+0000","last_fullsized":"2024-04-25T15:21:07.257566+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257299+0000","last_change":"2024-04-25T15:21:07.257299+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257299+0000","last_undegraded":"2024-04-25T15:21:07.257299+0000","last_fullsized":"2024-04-25T15:21:07.257299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261464+0000","last_change":"2024-04-25T15:21:07.261464+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261464+0000","last_undegraded":"2024-04-25T15:21:07.261464+0000","last_fullsized":"2024-04-25T15:21:07.261464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256914+0000","last_change":"2024-04-25T15:21:07.256914+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256914+0000","last_undegraded":"2024-04-25T15:21:07.256914+0000","last_fullsized":"2024-04-25T15:21:07.256914+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259855+0000","last_change":"2024-04-25T15:21:07.259855+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259855+0000","last_undegraded":"2024-04-25T15:21:07.259855+0000","last_fullsized":"2024-04-25T15:21:07.259855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252242+0000","last_change":"2024-04-25T15:21:07.252242+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252242+0000","last_undegraded":"2024-04-25T15:21:07.252242+0000","last_fullsized":"2024-04-25T15:21:07.252242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260409+0000","last_change":"2024-04-25T15:21:07.260409+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260409+0000","last_undegraded":"2024-04-25T15:21:07.260409+0000","last_fullsized":"2024-04-25T15:21:07.260409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259146+0000","last_change":"2024-04-25T15:21:07.259146+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259146+0000","last_undegraded":"2024-04-25T15:21:07.259146+0000","last_fullsized":"2024-04-25T15:21:07.259146+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255377+0000","last_change":"2024-04-25T15:21:07.255377+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255377+0000","last_undegraded":"2024-04-25T15:21:07.255377+0000","last_fullsized":"2024-04-25T15:21:07.255377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254666+0000","last_change":"2024-04-25T15:21:07.254666+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254666+0000","last_undegraded":"2024-04-25T15:21:07.254666+0000","last_fullsized":"2024-04-25T15:21:07.254666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257141+0000","last_change":"2024-04-25T15:21:07.257141+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257141+0000","last_undegraded":"2024-04-25T15:21:07.257141+0000","last_fullsized":"2024-04-25T15:21:07.257141+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277551+0000","last_change":"2024-04-25T15:21:07.273191+0000","last_active":"2024-04-25T15:21:07.277551+0000","last_peered":"2024-04-25T15:21:07.277551+0000","last_clean":"2024-04-25T15:21:07.277551+0000","last_became_active":"2024-04-25T15:21:07.272854+0000","last_became_peered":"2024-04-25T15:21:07.272854+0000","last_unstale":"2024-04-25T15:21:07.277551+0000","last_undegraded":"2024-04-25T15:21:07.277551+0000","last_fullsized":"2024-04-25T15:21:07.277551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:00.687305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257143+0000","last_change":"2024-04-25T15:21:07.257143+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257143+0000","last_undegraded":"2024-04-25T15:21:07.257143+0000","last_fullsized":"2024-04-25T15:21:07.257143+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270022+0000","last_change":"2024-04-25T15:21:07.268440+0000","last_active":"2024-04-25T15:21:07.270022+0000","last_peered":"2024-04-25T15:21:07.270022+0000","last_clean":"2024-04-25T15:21:07.270022+0000","last_became_active":"2024-04-25T15:21:07.268125+0000","last_became_peered":"2024-04-25T15:21:07.268125+0000","last_unstale":"2024-04-25T15:21:07.270022+0000","last_undegraded":"2024-04-25T15:21:07.270022+0000","last_fullsized":"2024-04-25T15:21:07.270022+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254113+0000","last_change":"2024-04-25T15:21:07.254113+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254113+0000","last_undegraded":"2024-04-25T15:21:07.254113+0000","last_fullsized":"2024-04-25T15:21:07.254113+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272789+0000","last_change":"2024-04-25T15:21:07.266826+0000","last_active":"2024-04-25T15:21:07.272789+0000","last_peered":"2024-04-25T15:21:07.272789+0000","last_clean":"2024-04-25T15:21:07.272789+0000","last_became_active":"2024-04-25T15:21:07.266540+0000","last_became_peered":"2024-04-25T15:21:07.266540+0000","last_unstale":"2024-04-25T15:21:07.272789+0000","last_undegraded":"2024-04-25T15:21:07.272789+0000","last_fullsized":"2024-04-25T15:21:07.272789+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258984+0000","last_change":"2024-04-25T15:21:07.258984+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258984+0000","last_undegraded":"2024-04-25T15:21:07.258984+0000","last_fullsized":"2024-04-25T15:21:07.258984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272412+0000","last_change":"2024-04-25T15:21:07.268165+0000","last_active":"2024-04-25T15:21:07.272412+0000","last_peered":"2024-04-25T15:21:07.272412+0000","last_clean":"2024-04-25T15:21:07.272412+0000","last_became_active":"2024-04-25T15:21:07.267821+0000","last_became_peered":"2024-04-25T15:21:07.267821+0000","last_unstale":"2024-04-25T15:21:07.272412+0000","last_undegraded":"2024-04-25T15:21:07.272412+0000","last_fullsized":"2024-04-25T15:21:07.272412+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255378+0000","last_change":"2024-04-25T15:21:07.255378+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255378+0000","last_undegraded":"2024-04-25T15:21:07.255378+0000","last_fullsized":"2024-04-25T15:21:07.255378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.260943+0000","last_change":"2024-04-25T15:21:07.258767+0000","last_active":"2024-04-25T15:21:07.260943+0000","last_peered":"2024-04-25T15:21:07.260943+0000","last_clean":"2024-04-25T15:21:07.260943+0000","last_became_active":"2024-04-25T15:21:07.258461+0000","last_became_peered":"2024-04-25T15:21:07.258461+0000","last_unstale":"2024-04-25T15:21:07.260943+0000","last_undegraded":"2024-04-25T15:21:07.260943+0000","last_fullsized":"2024-04-25T15:21:07.260943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252100+0000","last_change":"2024-04-25T15:21:07.252100+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252100+0000","last_undegraded":"2024-04-25T15:21:07.252100+0000","last_fullsized":"2024-04-25T15:21:07.252100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280453+0000","last_change":"2024-04-25T15:21:07.273310+0000","last_active":"2024-04-25T15:21:07.280453+0000","last_peered":"2024-04-25T15:21:07.280453+0000","last_clean":"2024-04-25T15:21:07.280453+0000","last_became_active":"2024-04-25T15:21:07.272977+0000","last_became_peered":"2024-04-25T15:21:07.272977+0000","last_unstale":"2024-04-25T15:21:07.280453+0000","last_undegraded":"2024-04-25T15:21:07.280453+0000","last_fullsized":"2024-04-25T15:21:07.280453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253544+0000","last_change":"2024-04-25T15:21:07.253544+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253544+0000","last_undegraded":"2024-04-25T15:21:07.253544+0000","last_fullsized":"2024-04-25T15:21:07.253544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.271664+0000","last_change":"2024-04-25T15:21:07.263740+0000","last_active":"2024-04-25T15:21:07.271664+0000","last_peered":"2024-04-25T15:21:07.271664+0000","last_clean":"2024-04-25T15:21:07.271664+0000","last_became_active":"2024-04-25T15:21:07.262783+0000","last_became_peered":"2024-04-25T15:21:07.262783+0000","last_unstale":"2024-04-25T15:21:07.271664+0000","last_undegraded":"2024-04-25T15:21:07.271664+0000","last_fullsized":"2024-04-25T15:21:07.271664+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258866+0000","last_change":"2024-04-25T15:21:07.258866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258866+0000","last_undegraded":"2024-04-25T15:21:07.258866+0000","last_fullsized":"2024-04-25T15:21:07.258866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272273+0000","last_change":"2024-04-25T15:21:07.267048+0000","last_active":"2024-04-25T15:21:07.272273+0000","last_peered":"2024-04-25T15:21:07.272273+0000","last_clean":"2024-04-25T15:21:07.272273+0000","last_became_active":"2024-04-25T15:21:07.265790+0000","last_became_peered":"2024-04-25T15:21:07.265790+0000","last_unstale":"2024-04-25T15:21:07.272273+0000","last_undegraded":"2024-04-25T15:21:07.272273+0000","last_fullsized":"2024-04-25T15:21:07.272273+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257928+0000","last_change":"2024-04-25T15:21:07.257928+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257928+0000","last_undegraded":"2024-04-25T15:21:07.257928+0000","last_fullsized":"2024-04-25T15:21:07.257928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279451+0000","last_change":"2024-04-25T15:21:07.273306+0000","last_active":"2024-04-25T15:21:07.279451+0000","last_peered":"2024-04-25T15:21:07.279451+0000","last_clean":"2024-04-25T15:21:07.279451+0000","last_became_active":"2024-04-25T15:21:07.272979+0000","last_became_peered":"2024-04-25T15:21:07.272979+0000","last_unstale":"2024-04-25T15:21:07.279451+0000","last_undegraded":"2024-04-25T15:21:07.279451+0000","last_fullsized":"2024-04-25T15:21:07.279451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:09:56.214534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.262262+0000","last_change":"2024-04-25T15:21:07.262262+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.262262+0000","last_undegraded":"2024-04-25T15:21:07.262262+0000","last_fullsized":"2024-04-25T15:21:07.262262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270661+0000","last_change":"2024-04-25T15:21:07.268388+0000","last_active":"2024-04-25T15:21:07.270661+0000","last_peered":"2024-04-25T15:21:07.270661+0000","last_clean":"2024-04-25T15:21:07.270661+0000","last_became_active":"2024-04-25T15:21:07.267977+0000","last_became_peered":"2024-04-25T15:21:07.267977+0000","last_unstale":"2024-04-25T15:21:07.270661+0000","last_undegraded":"2024-04-25T15:21:07.270661+0000","last_fullsized":"2024-04-25T15:21:07.270661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254905+0000","last_change":"2024-04-25T15:21:07.254905+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254905+0000","last_undegraded":"2024-04-25T15:21:07.254905+0000","last_fullsized":"2024-04-25T15:21:07.254905+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270271+0000","last_change":"2024-04-25T15:21:07.263141+0000","last_active":"2024-04-25T15:21:07.270271+0000","last_peered":"2024-04-25T15:21:07.270271+0000","last_clean":"2024-04-25T15:21:07.270271+0000","last_became_active":"2024-04-25T15:21:07.262765+0000","last_became_peered":"2024-04-25T15:21:07.262765+0000","last_unstale":"2024-04-25T15:21:07.270271+0000","last_undegraded":"2024-04-25T15:21:07.270271+0000","last_fullsized":"2024-04-25T15:21:07.270271+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252968+0000","last_change":"2024-04-25T15:21:07.252968+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252968+0000","last_undegraded":"2024-04-25T15:21:07.252968+0000","last_fullsized":"2024-04-25T15:21:07.252968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272311+0000","last_change":"2024-04-25T15:21:07.268503+0000","last_active":"2024-04-25T15:21:07.272311+0000","last_peered":"2024-04-25T15:21:07.272311+0000","last_clean":"2024-04-25T15:21:07.272311+0000","last_became_active":"2024-04-25T15:21:07.268088+0000","last_became_peered":"2024-04-25T15:21:07.268088+0000","last_unstale":"2024-04-25T15:21:07.272311+0000","last_undegraded":"2024-04-25T15:21:07.272311+0000","last_fullsized":"2024-04-25T15:21:07.272311+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:36.404924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254326+0000","last_change":"2024-04-25T15:21:07.254326+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254326+0000","last_undegraded":"2024-04-25T15:21:07.254326+0000","last_fullsized":"2024-04-25T15:21:07.254326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272165+0000","last_change":"2024-04-25T15:21:07.267053+0000","last_active":"2024-04-25T15:21:07.272165+0000","last_peered":"2024-04-25T15:21:07.272165+0000","last_clean":"2024-04-25T15:21:07.272165+0000","last_became_active":"2024-04-25T15:21:07.266690+0000","last_became_peered":"2024-04-25T15:21:07.266690+0000","last_unstale":"2024-04-25T15:21:07.272165+0000","last_undegraded":"2024-04-25T15:21:07.272165+0000","last_fullsized":"2024-04-25T15:21:07.272165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:47.899553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257171+0000","last_change":"2024-04-25T15:21:07.257171+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257171+0000","last_undegraded":"2024-04-25T15:21:07.257171+0000","last_fullsized":"2024-04-25T15:21:07.257171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272200+0000","last_change":"2024-04-25T15:21:07.262699+0000","last_active":"2024-04-25T15:21:07.272200+0000","last_peered":"2024-04-25T15:21:07.272200+0000","last_clean":"2024-04-25T15:21:07.272200+0000","last_became_active":"2024-04-25T15:21:07.262384+0000","last_became_peered":"2024-04-25T15:21:07.262384+0000","last_unstale":"2024-04-25T15:21:07.272200+0000","last_undegraded":"2024-04-25T15:21:07.272200+0000","last_fullsized":"2024-04-25T15:21:07.272200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256033+0000","last_change":"2024-04-25T15:21:07.256033+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256033+0000","last_undegraded":"2024-04-25T15:21:07.256033+0000","last_fullsized":"2024-04-25T15:21:07.256033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274736+0000","last_change":"2024-04-25T15:21:07.270561+0000","last_active":"2024-04-25T15:21:07.274736+0000","last_peered":"2024-04-25T15:21:07.274736+0000","last_clean":"2024-04-25T15:21:07.274736+0000","last_became_active":"2024-04-25T15:21:07.270106+0000","last_became_peered":"2024-04-25T15:21:07.270106+0000","last_unstale":"2024-04-25T15:21:07.274736+0000","last_undegraded":"2024-04-25T15:21:07.274736+0000","last_fullsized":"2024-04-25T15:21:07.274736+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:14:54.724610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256795+0000","last_change":"2024-04-25T15:21:07.256795+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256795+0000","last_undegraded":"2024-04-25T15:21:07.256795+0000","last_fullsized":"2024-04-25T15:21:07.256795+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280489+0000","last_change":"2024-04-25T15:21:07.273386+0000","last_active":"2024-04-25T15:21:07.280489+0000","last_peered":"2024-04-25T15:21:07.280489+0000","last_clean":"2024-04-25T15:21:07.280489+0000","last_became_active":"2024-04-25T15:21:07.272939+0000","last_became_peered":"2024-04-25T15:21:07.272939+0000","last_unstale":"2024-04-25T15:21:07.280489+0000","last_undegraded":"2024-04-25T15:21:07.280489+0000","last_fullsized":"2024-04-25T15:21:07.280489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259420+0000","last_change":"2024-04-25T15:21:07.259420+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259420+0000","last_undegraded":"2024-04-25T15:21:07.259420+0000","last_fullsized":"2024-04-25T15:21:07.259420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270390+0000","last_change":"2024-04-25T15:21:07.266432+0000","last_active":"2024-04-25T15:21:07.270390+0000","last_peered":"2024-04-25T15:21:07.270390+0000","last_clean":"2024-04-25T15:21:07.270390+0000","last_became_active":"2024-04-25T15:21:07.266089+0000","last_became_peered":"2024-04-25T15:21:07.266089+0000","last_unstale":"2024-04-25T15:21:07.270390+0000","last_undegraded":"2024-04-25T15:21:07.270390+0000","last_fullsized":"2024-04-25T15:21:07.270390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254229+0000","last_change":"2024-04-25T15:21:07.254229+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254229+0000","last_undegraded":"2024-04-25T15:21:07.254229+0000","last_fullsized":"2024-04-25T15:21:07.254229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269440+0000","last_change":"2024-04-25T15:21:07.262181+0000","last_active":"2024-04-25T15:21:07.269440+0000","last_peered":"2024-04-25T15:21:07.269440+0000","last_clean":"2024-04-25T15:21:07.269440+0000","last_became_active":"2024-04-25T15:21:07.261828+0000","last_became_peered":"2024-04-25T15:21:07.261828+0000","last_unstale":"2024-04-25T15:21:07.269440+0000","last_undegraded":"2024-04-25T15:21:07.269440+0000","last_fullsized":"2024-04-25T15:21:07.269440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256308+0000","last_change":"2024-04-25T15:21:07.256308+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256308+0000","last_undegraded":"2024-04-25T15:21:07.256308+0000","last_fullsized":"2024-04-25T15:21:07.256308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276901+0000","last_change":"2024-04-25T15:21:07.270193+0000","last_active":"2024-04-25T15:21:07.276901+0000","last_peered":"2024-04-25T15:21:07.276901+0000","last_clean":"2024-04-25T15:21:07.276901+0000","last_became_active":"2024-04-25T15:21:07.269650+0000","last_became_peered":"2024-04-25T15:21:07.269650+0000","last_unstale":"2024-04-25T15:21:07.276901+0000","last_undegraded":"2024-04-25T15:21:07.276901+0000","last_fullsized":"2024-04-25T15:21:07.276901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:17.140514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258937+0000","last_change":"2024-04-25T15:21:07.258937+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258937+0000","last_undegraded":"2024-04-25T15:21:07.258937+0000","last_fullsized":"2024-04-25T15:21:07.258937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.273489+0000","last_change":"2024-04-25T15:21:07.270403+0000","last_active":"2024-04-25T15:21:07.273489+0000","last_peered":"2024-04-25T15:21:07.273489+0000","last_clean":"2024-04-25T15:21:07.273489+0000","last_became_active":"2024-04-25T15:21:07.270082+0000","last_became_peered":"2024-04-25T15:21:07.270082+0000","last_unstale":"2024-04-25T15:21:07.273489+0000","last_undegraded":"2024-04-25T15:21:07.273489+0000","last_fullsized":"2024-04-25T15:21:07.273489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256563+0000","last_change":"2024-04-25T15:21:07.256563+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256563+0000","last_undegraded":"2024-04-25T15:21:07.256563+0000","last_fullsized":"2024-04-25T15:21:07.256563+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276978+0000","last_change":"2024-04-25T15:21:07.270988+0000","last_active":"2024-04-25T15:21:07.276978+0000","last_peered":"2024-04-25T15:21:07.276978+0000","last_clean":"2024-04-25T15:21:07.276978+0000","last_became_active":"2024-04-25T15:21:07.270686+0000","last_became_peered":"2024-04-25T15:21:07.270686+0000","last_unstale":"2024-04-25T15:21:07.276978+0000","last_undegraded":"2024-04-25T15:21:07.276978+0000","last_fullsized":"2024-04-25T15:21:07.276978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:14.556209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253688+0000","last_change":"2024-04-25T15:21:07.253688+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253688+0000","last_undegraded":"2024-04-25T15:21:07.253688+0000","last_fullsized":"2024-04-25T15:21:07.253688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276948+0000","last_change":"2024-04-25T15:21:07.270720+0000","last_active":"2024-04-25T15:21:07.276948+0000","last_peered":"2024-04-25T15:21:07.276948+0000","last_clean":"2024-04-25T15:21:07.276948+0000","last_became_active":"2024-04-25T15:21:07.270418+0000","last_became_peered":"2024-04-25T15:21:07.270418+0000","last_unstale":"2024-04-25T15:21:07.276948+0000","last_undegraded":"2024-04-25T15:21:07.276948+0000","last_fullsized":"2024-04-25T15:21:07.276948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:23.664366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254153+0000","last_change":"2024-04-25T15:21:07.254153+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254153+0000","last_undegraded":"2024-04-25T15:21:07.254153+0000","last_fullsized":"2024-04-25T15:21:07.254153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274957+0000","last_change":"2024-04-25T15:21:07.270169+0000","last_active":"2024-04-25T15:21:07.274957+0000","last_peered":"2024-04-25T15:21:07.274957+0000","last_clean":"2024-04-25T15:21:07.274957+0000","last_became_active":"2024-04-25T15:21:07.269753+0000","last_became_peered":"2024-04-25T15:21:07.269753+0000","last_unstale":"2024-04-25T15:21:07.274957+0000","last_undegraded":"2024-04-25T15:21:07.274957+0000","last_fullsized":"2024-04-25T15:21:07.274957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:37.213221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257535+0000","last_change":"2024-04-25T15:21:07.257535+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257535+0000","last_undegraded":"2024-04-25T15:21:07.257535+0000","last_fullsized":"2024-04-25T15:21:07.257535+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272859+0000","last_change":"2024-04-25T15:21:07.269932+0000","last_active":"2024-04-25T15:21:07.272859+0000","last_peered":"2024-04-25T15:21:07.272859+0000","last_clean":"2024-04-25T15:21:07.272859+0000","last_became_active":"2024-04-25T15:21:07.269598+0000","last_became_peered":"2024-04-25T15:21:07.269598+0000","last_unstale":"2024-04-25T15:21:07.272859+0000","last_undegraded":"2024-04-25T15:21:07.272859+0000","last_fullsized":"2024-04-25T15:21:07.272859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:39.459934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256124+0000","last_change":"2024-04-25T15:21:07.256124+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256124+0000","last_undegraded":"2024-04-25T15:21:07.256124+0000","last_fullsized":"2024-04-25T15:21:07.256124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270365+0000","last_change":"2024-04-25T15:21:07.263451+0000","last_active":"2024-04-25T15:21:07.270365+0000","last_peered":"2024-04-25T15:21:07.270365+0000","last_clean":"2024-04-25T15:21:07.270365+0000","last_became_active":"2024-04-25T15:21:07.263111+0000","last_became_peered":"2024-04-25T15:21:07.263111+0000","last_unstale":"2024-04-25T15:21:07.270365+0000","last_undegraded":"2024-04-25T15:21:07.270365+0000","last_fullsized":"2024-04-25T15:21:07.270365+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261818+0000","last_change":"2024-04-25T15:21:07.261818+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261818+0000","last_undegraded":"2024-04-25T15:21:07.261818+0000","last_fullsized":"2024-04-25T15:21:07.261818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277011+0000","last_change":"2024-04-25T15:21:07.271421+0000","last_active":"2024-04-25T15:21:07.277011+0000","last_peered":"2024-04-25T15:21:07.277011+0000","last_clean":"2024-04-25T15:21:07.277011+0000","last_became_active":"2024-04-25T15:21:07.271136+0000","last_became_peered":"2024-04-25T15:21:07.271136+0000","last_unstale":"2024-04-25T15:21:07.277011+0000","last_undegraded":"2024-04-25T15:21:07.277011+0000","last_fullsized":"2024-04-25T15:21:07.277011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:34.931594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258775+0000","last_change":"2024-04-25T15:21:07.258775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258775+0000","last_undegraded":"2024-04-25T15:21:07.258775+0000","last_fullsized":"2024-04-25T15:21:07.258775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269969+0000","last_change":"2024-04-25T15:21:07.267201+0000","last_active":"2024-04-25T15:21:07.269969+0000","last_peered":"2024-04-25T15:21:07.269969+0000","last_clean":"2024-04-25T15:21:07.269969+0000","last_became_active":"2024-04-25T15:21:07.266901+0000","last_became_peered":"2024-04-25T15:21:07.266901+0000","last_unstale":"2024-04-25T15:21:07.269969+0000","last_undegraded":"2024-04-25T15:21:07.269969+0000","last_fullsized":"2024-04-25T15:21:07.269969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253682+0000","last_change":"2024-04-25T15:21:07.253682+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253682+0000","last_undegraded":"2024-04-25T15:21:07.253682+0000","last_fullsized":"2024-04-25T15:21:07.253682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270067+0000","last_change":"2024-04-25T15:21:07.265869+0000","last_active":"2024-04-25T15:21:07.270067+0000","last_peered":"2024-04-25T15:21:07.270067+0000","last_clean":"2024-04-25T15:21:07.270067+0000","last_became_active":"2024-04-25T15:21:07.265566+0000","last_became_peered":"2024-04-25T15:21:07.265566+0000","last_unstale":"2024-04-25T15:21:07.270067+0000","last_undegraded":"2024-04-25T15:21:07.270067+0000","last_fullsized":"2024-04-25T15:21:07.270067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256908+0000","last_change":"2024-04-25T15:21:07.256908+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256908+0000","last_undegraded":"2024-04-25T15:21:07.256908+0000","last_fullsized":"2024-04-25T15:21:07.256908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272141+0000","last_change":"2024-04-25T15:21:07.267132+0000","last_active":"2024-04-25T15:21:07.272141+0000","last_peered":"2024-04-25T15:21:07.272141+0000","last_clean":"2024-04-25T15:21:07.272141+0000","last_became_active":"2024-04-25T15:21:07.266783+0000","last_became_peered":"2024-04-25T15:21:07.266783+0000","last_unstale":"2024-04-25T15:21:07.272141+0000","last_undegraded":"2024-04-25T15:21:07.272141+0000","last_fullsized":"2024-04-25T15:21:07.272141+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:10.461067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255866+0000","last_change":"2024-04-25T15:21:07.255866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255866+0000","last_undegraded":"2024-04-25T15:21:07.255866+0000","last_fullsized":"2024-04-25T15:21:07.255866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272473+0000","last_change":"2024-04-25T15:21:07.268347+0000","last_active":"2024-04-25T15:21:07.272473+0000","last_peered":"2024-04-25T15:21:07.272473+0000","last_clean":"2024-04-25T15:21:07.272473+0000","last_became_active":"2024-04-25T15:21:07.268032+0000","last_became_peered":"2024-04-25T15:21:07.268032+0000","last_unstale":"2024-04-25T15:21:07.272473+0000","last_undegraded":"2024-04-25T15:21:07.272473+0000","last_fullsized":"2024-04-25T15:21:07.272473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260480+0000","last_change":"2024-04-25T15:21:07.260480+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260480+0000","last_undegraded":"2024-04-25T15:21:07.260480+0000","last_fullsized":"2024-04-25T15:21:07.260480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279438+0000","last_change":"2024-04-25T15:21:07.272594+0000","last_active":"2024-04-25T15:21:07.279438+0000","last_peered":"2024-04-25T15:21:07.279438+0000","last_clean":"2024-04-25T15:21:07.279438+0000","last_became_active":"2024-04-25T15:21:07.272274+0000","last_became_peered":"2024-04-25T15:21:07.272274+0000","last_unstale":"2024-04-25T15:21:07.279438+0000","last_undegraded":"2024-04-25T15:21:07.279438+0000","last_fullsized":"2024-04-25T15:21:07.279438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:09.551367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254935+0000","last_change":"2024-04-25T15:21:07.254935+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254935+0000","last_undegraded":"2024-04-25T15:21:07.254935+0000","last_fullsized":"2024-04-25T15:21:07.254935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274974+0000","last_change":"2024-04-25T15:21:07.271612+0000","last_active":"2024-04-25T15:21:07.274974+0000","last_peered":"2024-04-25T15:21:07.274974+0000","last_clean":"2024-04-25T15:21:07.274974+0000","last_became_active":"2024-04-25T15:21:07.271320+0000","last_became_peered":"2024-04-25T15:21:07.271320+0000","last_unstale":"2024-04-25T15:21:07.274974+0000","last_undegraded":"2024-04-25T15:21:07.274974+0000","last_fullsized":"2024-04-25T15:21:07.274974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:42.520351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":141,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.259258+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:21:07.259258+0000","last_peered":"2024-04-25T15:21:07.259258+0000","last_clean":"2024-04-25T15:21:07.259258+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:21:07.259258+0000","last_undegraded":"2024-04-25T15:21:07.259258+0000","last_fullsized":"2024-04-25T15:21:07.259258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1019904,"data_stored":773544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:21:12.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:21:12.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.791+0000 7f8ef01c5640 1 Processor -- start 2024-04-25T15:21:12.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.791+0000 7f8ef01c5640 1 -- start start 2024-04-25T15:21:12.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.792+0000 7f8ef01c5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8ee80d0730 0x7f8ee80d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.792+0000 7f8ef01c5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8ee806ccc0 0x7f8ee806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.792+0000 7f8ef01c5640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee80b9200 con 0x7f8ee80d4f60 2024-04-25T15:21:12.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.792+0000 7f8ef01c5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee806c550 con 0x7f8ee80d0730 2024-04-25T15:21:12.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.792+0000 7f8ef01c5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee806c830 con 0x7f8ee806ccc0 2024-04-25T15:21:12.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8ee80d0730 0x7f8ee80d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8ee80d0730 0x7f8ee80d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38112/0 (socket says 172.21.15.146:38112) 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 -- 172.21.15.146:0/1759532896 learned_addr learned my addr 172.21.15.146:0/1759532896 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eee762640 1 --2- 172.21.15.146:0/1759532896 >> v2:172.21.15.177:3300/0 conn(0x7f8ee806ccc0 0x7f8ee806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 -- 172.21.15.146:0/1759532896 >> v2:172.21.15.177:3300/0 conn(0x7f8ee806ccc0 msgr2=0x7f8ee806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 --2- 172.21.15.146:0/1759532896 >> v2:172.21.15.177:3300/0 conn(0x7f8ee806ccc0 0x7f8ee806d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 -- 172.21.15.146:0/1759532896 >> v1:172.21.15.177:6789/0 conn(0x7f8ee80d4f60 legacy=0x7f8ee80d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.793+0000 7f8eed760640 1 -- 172.21.15.146:0/1759532896 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee80d1070 con 0x7f8ee80d0730 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8eed760640 1 --2- 172.21.15.146:0/1759532896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d0730 0x7f8ee80d5370 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8ee80d3e10 tx=0x7f8ed802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d790a04951d75f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8eecf5f640 1 -- 172.21.15.146:0/1759532896 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8037070 con 0x7f8ee80d0730 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8eecf5f640 1 -- 172.21.15.146:0/1759532896 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed802ae20 con 0x7f8ee80d0730 2024-04-25T15:21:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8eecf5f640 1 -- 172.21.15.146:0/1759532896 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8032660 con 0x7f8ee80d0730 2024-04-25T15:21:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1759532896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d0730 msgr2=0x7f8ee80d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1759532896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d0730 0x7f8ee80d5370 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8ee80d3e10 tx=0x7f8ed802a110 comp rx=0 tx=0).stop 2024-04-25T15:21:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1759532896 shutdown_connections 2024-04-25T15:21:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1759532896 >> v2:172.21.15.177:3300/0 conn(0x7f8ee806ccc0 0x7f8ee806d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1759532896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d0730 0x7f8ee80d5370 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.794+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1759532896 >> 172.21.15.146:0/1759532896 conn(0x7f8ee80d2450 msgr2=0x7f8ee80d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:12.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.795+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1759532896 shutdown_connections 2024-04-25T15:21:12.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.795+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1759532896 wait complete. 2024-04-25T15:21:12.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.796+0000 7f8ef01c5640 1 Processor -- start 2024-04-25T15:21:12.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.796+0000 7f8ef01c5640 1 -- start start 2024-04-25T15:21:12.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ee806ccc0 0x7f8ee815f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 0x7f8ee815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ee815cfc0 0x7f8ee815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee80b5150 con 0x7f8ee80d4f60 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee80d15f0 con 0x7f8ee806ccc0 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee80b52d0 con 0x7f8ee815cfc0 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 0x7f8ee815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 0x7f8ee815e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38124/0 (socket says 172.21.15.146:38124) 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 -- 172.21.15.146:0/1133922745 learned_addr learned my addr 172.21.15.146:0/1133922745 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eee762640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ee815cfc0 0x7f8ee815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eedf61640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ee806ccc0 0x7f8ee815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 -- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ee815cfc0 msgr2=0x7f8ee815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ee815cfc0 0x7f8ee815ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 -- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ee806ccc0 msgr2=0x7f8ee815f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ee806ccc0 0x7f8ee815f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 -- 172.21.15.146:0/1133922745 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee8162fb0 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eedf61640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ee806ccc0 0x7f8ee815f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8eed760640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 0x7f8ee815e7b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8ed802a150 tx=0x7f8ed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8042070 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed803e670 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee815d710 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8037040 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.798+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee815dcd0 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.799+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee806a0e0 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.800+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8ed8045070 con 0x7f8ee80d4f60 2024-04-25T15:21:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.800+0000 7f8ede7fc640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8ecc0744b0 0x7f8ecc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.800+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8ed80b5a80 con 0x7f8ee80d4f60 2024-04-25T15:21:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.800+0000 7f8eedf61640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8ecc0744b0 0x7f8ecc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.801+0000 7f8eedf61640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8ecc0744b0 0x7f8ecc0769c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ee80b4880 tx=0x7f8ee4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:12.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.805+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ee80b9200 con 0x7f8ee80d4f60 2024-04-25T15:21:12.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.977+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ee8163f20 con 0x7f8ecc0744b0 2024-04-25T15:21:12.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.982+0000 7f8ede7fc640 1 -- 172.21.15.146:0/1133922745 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246623 (secure 0 0 0) 0x7f8ee8163f20 con 0x7f8ecc0744b0 2024-04-25T15:21:12.986 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8ecc0744b0 msgr2=0x7f8ecc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8ecc0744b0 0x7f8ecc0769c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8ee80b4880 tx=0x7f8ee4008040 comp rx=0 tx=0).stop 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 msgr2=0x7f8ee815e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 0x7f8ee815e7b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8ed802a150 tx=0x7f8ed80046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 shutdown_connections 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8ecc0744b0 0x7f8ecc0769c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ee815cfc0 0x7f8ee815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ee80d4f60 0x7f8ee815e7b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 --2- 172.21.15.146:0/1133922745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ee806ccc0 0x7f8ee815f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.985+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 >> 172.21.15.146:0/1133922745 conn(0x7f8ee80d2450 msgr2=0x7f8ee80ba630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.986+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 shutdown_connections 2024-04-25T15:21:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:12.986+0000 7f8ef01c5640 1 -- 172.21.15.146:0/1133922745 wait complete. 2024-04-25T15:21:12.997 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-25T15:21:12.005640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220644,"kb_used_data":4464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754076,"statfs":{"total":773094113280,"available":772868173824,"internally_reserved":0,"allocated":4571136,"data_stored":2632810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758904"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258465+0000","last_change":"2024-04-25T15:21:07.258465+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258465+0000","last_undegraded":"2024-04-25T15:21:07.258465+0000","last_fullsized":"2024-04-25T15:21:07.258465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259346+0000","last_change":"2024-04-25T15:21:07.259346+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259346+0000","last_undegraded":"2024-04-25T15:21:07.259346+0000","last_fullsized":"2024-04-25T15:21:07.259346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.265056+0000","last_change":"2024-04-25T15:21:07.265056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.265056+0000","last_undegraded":"2024-04-25T15:21:07.265056+0000","last_fullsized":"2024-04-25T15:21:07.265056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258425+0000","last_change":"2024-04-25T15:21:07.258425+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258425+0000","last_undegraded":"2024-04-25T15:21:07.258425+0000","last_fullsized":"2024-04-25T15:21:07.258425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260423+0000","last_change":"2024-04-25T15:21:07.260423+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260423+0000","last_undegraded":"2024-04-25T15:21:07.260423+0000","last_fullsized":"2024-04-25T15:21:07.260423+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256056+0000","last_change":"2024-04-25T15:21:07.256056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256056+0000","last_undegraded":"2024-04-25T15:21:07.256056+0000","last_fullsized":"2024-04-25T15:21:07.256056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253502+0000","last_change":"2024-04-25T15:21:07.253502+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253502+0000","last_undegraded":"2024-04-25T15:21:07.253502+0000","last_fullsized":"2024-04-25T15:21:07.253502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259316+0000","last_change":"2024-04-25T15:21:07.259316+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259316+0000","last_undegraded":"2024-04-25T15:21:07.259316+0000","last_fullsized":"2024-04-25T15:21:07.259316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.264325+0000","last_change":"2024-04-25T15:21:07.264325+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.264325+0000","last_undegraded":"2024-04-25T15:21:07.264325+0000","last_fullsized":"2024-04-25T15:21:07.264325+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252907+0000","last_change":"2024-04-25T15:21:07.252907+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252907+0000","last_undegraded":"2024-04-25T15:21:07.252907+0000","last_fullsized":"2024-04-25T15:21:07.252907+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256157+0000","last_change":"2024-04-25T15:21:07.256157+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256157+0000","last_undegraded":"2024-04-25T15:21:07.256157+0000","last_fullsized":"2024-04-25T15:21:07.256157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255957+0000","last_change":"2024-04-25T15:21:07.255957+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255957+0000","last_undegraded":"2024-04-25T15:21:07.255957+0000","last_fullsized":"2024-04-25T15:21:07.255957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255463+0000","last_change":"2024-04-25T15:21:07.255463+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255463+0000","last_undegraded":"2024-04-25T15:21:07.255463+0000","last_fullsized":"2024-04-25T15:21:07.255463+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257032+0000","last_change":"2024-04-25T15:21:07.257032+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257032+0000","last_undegraded":"2024-04-25T15:21:07.257032+0000","last_fullsized":"2024-04-25T15:21:07.257032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252787+0000","last_change":"2024-04-25T15:21:07.252787+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252787+0000","last_undegraded":"2024-04-25T15:21:07.252787+0000","last_fullsized":"2024-04-25T15:21:07.252787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258605+0000","last_change":"2024-04-25T15:21:07.258605+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258605+0000","last_undegraded":"2024-04-25T15:21:07.258605+0000","last_fullsized":"2024-04-25T15:21:07.258605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260477+0000","last_change":"2024-04-25T15:21:07.260477+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260477+0000","last_undegraded":"2024-04-25T15:21:07.260477+0000","last_fullsized":"2024-04-25T15:21:07.260477+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260775+0000","last_change":"2024-04-25T15:21:07.260775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260775+0000","last_undegraded":"2024-04-25T15:21:07.260775+0000","last_fullsized":"2024-04-25T15:21:07.260775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261610+0000","last_change":"2024-04-25T15:21:07.261610+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261610+0000","last_undegraded":"2024-04-25T15:21:07.261610+0000","last_fullsized":"2024-04-25T15:21:07.261610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260690+0000","last_change":"2024-04-25T15:21:07.260690+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260690+0000","last_undegraded":"2024-04-25T15:21:07.260690+0000","last_fullsized":"2024-04-25T15:21:07.260690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254693+0000","last_change":"2024-04-25T15:21:07.254693+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254693+0000","last_undegraded":"2024-04-25T15:21:07.254693+0000","last_fullsized":"2024-04-25T15:21:07.254693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255362+0000","last_change":"2024-04-25T15:21:07.255362+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255362+0000","last_undegraded":"2024-04-25T15:21:07.255362+0000","last_fullsized":"2024-04-25T15:21:07.255362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257566+0000","last_change":"2024-04-25T15:21:07.257566+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257566+0000","last_undegraded":"2024-04-25T15:21:07.257566+0000","last_fullsized":"2024-04-25T15:21:07.257566+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257299+0000","last_change":"2024-04-25T15:21:07.257299+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257299+0000","last_undegraded":"2024-04-25T15:21:07.257299+0000","last_fullsized":"2024-04-25T15:21:07.257299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261464+0000","last_change":"2024-04-25T15:21:07.261464+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261464+0000","last_undegraded":"2024-04-25T15:21:07.261464+0000","last_fullsized":"2024-04-25T15:21:07.261464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256914+0000","last_change":"2024-04-25T15:21:07.256914+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256914+0000","last_undegraded":"2024-04-25T15:21:07.256914+0000","last_fullsized":"2024-04-25T15:21:07.256914+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259855+0000","last_change":"2024-04-25T15:21:07.259855+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259855+0000","last_undegraded":"2024-04-25T15:21:07.259855+0000","last_fullsized":"2024-04-25T15:21:07.259855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252242+0000","last_change":"2024-04-25T15:21:07.252242+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252242+0000","last_undegraded":"2024-04-25T15:21:07.252242+0000","last_fullsized":"2024-04-25T15:21:07.252242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260409+0000","last_change":"2024-04-25T15:21:07.260409+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260409+0000","last_undegraded":"2024-04-25T15:21:07.260409+0000","last_fullsized":"2024-04-25T15:21:07.260409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259146+0000","last_change":"2024-04-25T15:21:07.259146+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259146+0000","last_undegraded":"2024-04-25T15:21:07.259146+0000","last_fullsized":"2024-04-25T15:21:07.259146+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255377+0000","last_change":"2024-04-25T15:21:07.255377+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255377+0000","last_undegraded":"2024-04-25T15:21:07.255377+0000","last_fullsized":"2024-04-25T15:21:07.255377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254666+0000","last_change":"2024-04-25T15:21:07.254666+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254666+0000","last_undegraded":"2024-04-25T15:21:07.254666+0000","last_fullsized":"2024-04-25T15:21:07.254666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257141+0000","last_change":"2024-04-25T15:21:07.257141+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257141+0000","last_undegraded":"2024-04-25T15:21:07.257141+0000","last_fullsized":"2024-04-25T15:21:07.257141+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277551+0000","last_change":"2024-04-25T15:21:07.273191+0000","last_active":"2024-04-25T15:21:07.277551+0000","last_peered":"2024-04-25T15:21:07.277551+0000","last_clean":"2024-04-25T15:21:07.277551+0000","last_became_active":"2024-04-25T15:21:07.272854+0000","last_became_peered":"2024-04-25T15:21:07.272854+0000","last_unstale":"2024-04-25T15:21:07.277551+0000","last_undegraded":"2024-04-25T15:21:07.277551+0000","last_fullsized":"2024-04-25T15:21:07.277551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:00.687305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257143+0000","last_change":"2024-04-25T15:21:07.257143+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257143+0000","last_undegraded":"2024-04-25T15:21:07.257143+0000","last_fullsized":"2024-04-25T15:21:07.257143+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270022+0000","last_change":"2024-04-25T15:21:07.268440+0000","last_active":"2024-04-25T15:21:07.270022+0000","last_peered":"2024-04-25T15:21:07.270022+0000","last_clean":"2024-04-25T15:21:07.270022+0000","last_became_active":"2024-04-25T15:21:07.268125+0000","last_became_peered":"2024-04-25T15:21:07.268125+0000","last_unstale":"2024-04-25T15:21:07.270022+0000","last_undegraded":"2024-04-25T15:21:07.270022+0000","last_fullsized":"2024-04-25T15:21:07.270022+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254113+0000","last_change":"2024-04-25T15:21:07.254113+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254113+0000","last_undegraded":"2024-04-25T15:21:07.254113+0000","last_fullsized":"2024-04-25T15:21:07.254113+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272789+0000","last_change":"2024-04-25T15:21:07.266826+0000","last_active":"2024-04-25T15:21:07.272789+0000","last_peered":"2024-04-25T15:21:07.272789+0000","last_clean":"2024-04-25T15:21:07.272789+0000","last_became_active":"2024-04-25T15:21:07.266540+0000","last_became_peered":"2024-04-25T15:21:07.266540+0000","last_unstale":"2024-04-25T15:21:07.272789+0000","last_undegraded":"2024-04-25T15:21:07.272789+0000","last_fullsized":"2024-04-25T15:21:07.272789+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258984+0000","last_change":"2024-04-25T15:21:07.258984+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258984+0000","last_undegraded":"2024-04-25T15:21:07.258984+0000","last_fullsized":"2024-04-25T15:21:07.258984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272412+0000","last_change":"2024-04-25T15:21:07.268165+0000","last_active":"2024-04-25T15:21:07.272412+0000","last_peered":"2024-04-25T15:21:07.272412+0000","last_clean":"2024-04-25T15:21:07.272412+0000","last_became_active":"2024-04-25T15:21:07.267821+0000","last_became_peered":"2024-04-25T15:21:07.267821+0000","last_unstale":"2024-04-25T15:21:07.272412+0000","last_undegraded":"2024-04-25T15:21:07.272412+0000","last_fullsized":"2024-04-25T15:21:07.272412+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255378+0000","last_change":"2024-04-25T15:21:07.255378+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255378+0000","last_undegraded":"2024-04-25T15:21:07.255378+0000","last_fullsized":"2024-04-25T15:21:07.255378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.260943+0000","last_change":"2024-04-25T15:21:07.258767+0000","last_active":"2024-04-25T15:21:07.260943+0000","last_peered":"2024-04-25T15:21:07.260943+0000","last_clean":"2024-04-25T15:21:07.260943+0000","last_became_active":"2024-04-25T15:21:07.258461+0000","last_became_peered":"2024-04-25T15:21:07.258461+0000","last_unstale":"2024-04-25T15:21:07.260943+0000","last_undegraded":"2024-04-25T15:21:07.260943+0000","last_fullsized":"2024-04-25T15:21:07.260943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252100+0000","last_change":"2024-04-25T15:21:07.252100+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252100+0000","last_undegraded":"2024-04-25T15:21:07.252100+0000","last_fullsized":"2024-04-25T15:21:07.252100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280453+0000","last_change":"2024-04-25T15:21:07.273310+0000","last_active":"2024-04-25T15:21:07.280453+0000","last_peered":"2024-04-25T15:21:07.280453+0000","last_clean":"2024-04-25T15:21:07.280453+0000","last_became_active":"2024-04-25T15:21:07.272977+0000","last_became_peered":"2024-04-25T15:21:07.272977+0000","last_unstale":"2024-04-25T15:21:07.280453+0000","last_undegraded":"2024-04-25T15:21:07.280453+0000","last_fullsized":"2024-04-25T15:21:07.280453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253544+0000","last_change":"2024-04-25T15:21:07.253544+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253544+0000","last_undegraded":"2024-04-25T15:21:07.253544+0000","last_fullsized":"2024-04-25T15:21:07.253544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.271664+0000","last_change":"2024-04-25T15:21:07.263740+0000","last_active":"2024-04-25T15:21:07.271664+0000","last_peered":"2024-04-25T15:21:07.271664+0000","last_clean":"2024-04-25T15:21:07.271664+0000","last_became_active":"2024-04-25T15:21:07.262783+0000","last_became_peered":"2024-04-25T15:21:07.262783+0000","last_unstale":"2024-04-25T15:21:07.271664+0000","last_undegraded":"2024-04-25T15:21:07.271664+0000","last_fullsized":"2024-04-25T15:21:07.271664+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258866+0000","last_change":"2024-04-25T15:21:07.258866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258866+0000","last_undegraded":"2024-04-25T15:21:07.258866+0000","last_fullsized":"2024-04-25T15:21:07.258866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272273+0000","last_change":"2024-04-25T15:21:07.267048+0000","last_active":"2024-04-25T15:21:07.272273+0000","last_peered":"2024-04-25T15:21:07.272273+0000","last_clean":"2024-04-25T15:21:07.272273+0000","last_became_active":"2024-04-25T15:21:07.265790+0000","last_became_peered":"2024-04-25T15:21:07.265790+0000","last_unstale":"2024-04-25T15:21:07.272273+0000","last_undegraded":"2024-04-25T15:21:07.272273+0000","last_fullsized":"2024-04-25T15:21:07.272273+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257928+0000","last_change":"2024-04-25T15:21:07.257928+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257928+0000","last_undegraded":"2024-04-25T15:21:07.257928+0000","last_fullsized":"2024-04-25T15:21:07.257928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279451+0000","last_change":"2024-04-25T15:21:07.273306+0000","last_active":"2024-04-25T15:21:07.279451+0000","last_peered":"2024-04-25T15:21:07.279451+0000","last_clean":"2024-04-25T15:21:07.279451+0000","last_became_active":"2024-04-25T15:21:07.272979+0000","last_became_peered":"2024-04-25T15:21:07.272979+0000","last_unstale":"2024-04-25T15:21:07.279451+0000","last_undegraded":"2024-04-25T15:21:07.279451+0000","last_fullsized":"2024-04-25T15:21:07.279451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:09:56.214534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.262262+0000","last_change":"2024-04-25T15:21:07.262262+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.262262+0000","last_undegraded":"2024-04-25T15:21:07.262262+0000","last_fullsized":"2024-04-25T15:21:07.262262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270661+0000","last_change":"2024-04-25T15:21:07.268388+0000","last_active":"2024-04-25T15:21:07.270661+0000","last_peered":"2024-04-25T15:21:07.270661+0000","last_clean":"2024-04-25T15:21:07.270661+0000","last_became_active":"2024-04-25T15:21:07.267977+0000","last_became_peered":"2024-04-25T15:21:07.267977+0000","last_unstale":"2024-04-25T15:21:07.270661+0000","last_undegraded":"2024-04-25T15:21:07.270661+0000","last_fullsized":"2024-04-25T15:21:07.270661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254905+0000","last_change":"2024-04-25T15:21:07.254905+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254905+0000","last_undegraded":"2024-04-25T15:21:07.254905+0000","last_fullsized":"2024-04-25T15:21:07.254905+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270271+0000","last_change":"2024-04-25T15:21:07.263141+0000","last_active":"2024-04-25T15:21:07.270271+0000","last_peered":"2024-04-25T15:21:07.270271+0000","last_clean":"2024-04-25T15:21:07.270271+0000","last_became_active":"2024-04-25T15:21:07.262765+0000","last_became_peered":"2024-04-25T15:21:07.262765+0000","last_unstale":"2024-04-25T15:21:07.270271+0000","last_undegraded":"2024-04-25T15:21:07.270271+0000","last_fullsized":"2024-04-25T15:21:07.270271+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252968+0000","last_change":"2024-04-25T15:21:07.252968+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252968+0000","last_undegraded":"2024-04-25T15:21:07.252968+0000","last_fullsized":"2024-04-25T15:21:07.252968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272311+0000","last_change":"2024-04-25T15:21:07.268503+0000","last_active":"2024-04-25T15:21:07.272311+0000","last_peered":"2024-04-25T15:21:07.272311+0000","last_clean":"2024-04-25T15:21:07.272311+0000","last_became_active":"2024-04-25T15:21:07.268088+0000","last_became_peered":"2024-04-25T15:21:07.268088+0000","last_unstale":"2024-04-25T15:21:07.272311+0000","last_undegraded":"2024-04-25T15:21:07.272311+0000","last_fullsized":"2024-04-25T15:21:07.272311+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:36.404924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254326+0000","last_change":"2024-04-25T15:21:07.254326+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254326+0000","last_undegraded":"2024-04-25T15:21:07.254326+0000","last_fullsized":"2024-04-25T15:21:07.254326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272165+0000","last_change":"2024-04-25T15:21:07.267053+0000","last_active":"2024-04-25T15:21:07.272165+0000","last_peered":"2024-04-25T15:21:07.272165+0000","last_clean":"2024-04-25T15:21:07.272165+0000","last_became_active":"2024-04-25T15:21:07.266690+0000","last_became_peered":"2024-04-25T15:21:07.266690+0000","last_unstale":"2024-04-25T15:21:07.272165+0000","last_undegraded":"2024-04-25T15:21:07.272165+0000","last_fullsized":"2024-04-25T15:21:07.272165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:47.899553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257171+0000","last_change":"2024-04-25T15:21:07.257171+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257171+0000","last_undegraded":"2024-04-25T15:21:07.257171+0000","last_fullsized":"2024-04-25T15:21:07.257171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272200+0000","last_change":"2024-04-25T15:21:07.262699+0000","last_active":"2024-04-25T15:21:07.272200+0000","last_peered":"2024-04-25T15:21:07.272200+0000","last_clean":"2024-04-25T15:21:07.272200+0000","last_became_active":"2024-04-25T15:21:07.262384+0000","last_became_peered":"2024-04-25T15:21:07.262384+0000","last_unstale":"2024-04-25T15:21:07.272200+0000","last_undegraded":"2024-04-25T15:21:07.272200+0000","last_fullsized":"2024-04-25T15:21:07.272200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256033+0000","last_change":"2024-04-25T15:21:07.256033+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256033+0000","last_undegraded":"2024-04-25T15:21:07.256033+0000","last_fullsized":"2024-04-25T15:21:07.256033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274736+0000","last_change":"2024-04-25T15:21:07.270561+0000","last_active":"2024-04-25T15:21:07.274736+0000","last_peered":"2024-04-25T15:21:07.274736+0000","last_clean":"2024-04-25T15:21:07.274736+0000","last_became_active":"2024-04-25T15:21:07.270106+0000","last_became_peered":"2024-04-25T15:21:07.270106+0000","last_unstale":"2024-04-25T15:21:07.274736+0000","last_undegraded":"2024-04-25T15:21:07.274736+0000","last_fullsized":"2024-04-25T15:21:07.274736+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:14:54.724610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256795+0000","last_change":"2024-04-25T15:21:07.256795+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256795+0000","last_undegraded":"2024-04-25T15:21:07.256795+0000","last_fullsized":"2024-04-25T15:21:07.256795+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280489+0000","last_change":"2024-04-25T15:21:07.273386+0000","last_active":"2024-04-25T15:21:07.280489+0000","last_peered":"2024-04-25T15:21:07.280489+0000","last_clean":"2024-04-25T15:21:07.280489+0000","last_became_active":"2024-04-25T15:21:07.272939+0000","last_became_peered":"2024-04-25T15:21:07.272939+0000","last_unstale":"2024-04-25T15:21:07.280489+0000","last_undegraded":"2024-04-25T15:21:07.280489+0000","last_fullsized":"2024-04-25T15:21:07.280489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259420+0000","last_change":"2024-04-25T15:21:07.259420+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259420+0000","last_undegraded":"2024-04-25T15:21:07.259420+0000","last_fullsized":"2024-04-25T15:21:07.259420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270390+0000","last_change":"2024-04-25T15:21:07.266432+0000","last_active":"2024-04-25T15:21:07.270390+0000","last_peered":"2024-04-25T15:21:07.270390+0000","last_clean":"2024-04-25T15:21:07.270390+0000","last_became_active":"2024-04-25T15:21:07.266089+0000","last_became_peered":"2024-04-25T15:21:07.266089+0000","last_unstale":"2024-04-25T15:21:07.270390+0000","last_undegraded":"2024-04-25T15:21:07.270390+0000","last_fullsized":"2024-04-25T15:21:07.270390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254229+0000","last_change":"2024-04-25T15:21:07.254229+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254229+0000","last_undegraded":"2024-04-25T15:21:07.254229+0000","last_fullsized":"2024-04-25T15:21:07.254229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269440+0000","last_change":"2024-04-25T15:21:07.262181+0000","last_active":"2024-04-25T15:21:07.269440+0000","last_peered":"2024-04-25T15:21:07.269440+0000","last_clean":"2024-04-25T15:21:07.269440+0000","last_became_active":"2024-04-25T15:21:07.261828+0000","last_became_peered":"2024-04-25T15:21:07.261828+0000","last_unstale":"2024-04-25T15:21:07.269440+0000","last_undegraded":"2024-04-25T15:21:07.269440+0000","last_fullsized":"2024-04-25T15:21:07.269440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256308+0000","last_change":"2024-04-25T15:21:07.256308+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256308+0000","last_undegraded":"2024-04-25T15:21:07.256308+0000","last_fullsized":"2024-04-25T15:21:07.256308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276901+0000","last_change":"2024-04-25T15:21:07.270193+0000","last_active":"2024-04-25T15:21:07.276901+0000","last_peered":"2024-04-25T15:21:07.276901+0000","last_clean":"2024-04-25T15:21:07.276901+0000","last_became_active":"2024-04-25T15:21:07.269650+0000","last_became_peered":"2024-04-25T15:21:07.269650+0000","last_unstale":"2024-04-25T15:21:07.276901+0000","last_undegraded":"2024-04-25T15:21:07.276901+0000","last_fullsized":"2024-04-25T15:21:07.276901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:17.140514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258937+0000","last_change":"2024-04-25T15:21:07.258937+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258937+0000","last_undegraded":"2024-04-25T15:21:07.258937+0000","last_fullsized":"2024-04-25T15:21:07.258937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.273489+0000","last_change":"2024-04-25T15:21:07.270403+0000","last_active":"2024-04-25T15:21:07.273489+0000","last_peered":"2024-04-25T15:21:07.273489+0000","last_clean":"2024-04-25T15:21:07.273489+0000","last_became_active":"2024-04-25T15:21:07.270082+0000","last_became_peered":"2024-04-25T15:21:07.270082+0000","last_unstale":"2024-04-25T15:21:07.273489+0000","last_undegraded":"2024-04-25T15:21:07.273489+0000","last_fullsized":"2024-04-25T15:21:07.273489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256563+0000","last_change":"2024-04-25T15:21:07.256563+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256563+0000","last_undegraded":"2024-04-25T15:21:07.256563+0000","last_fullsized":"2024-04-25T15:21:07.256563+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276978+0000","last_change":"2024-04-25T15:21:07.270988+0000","last_active":"2024-04-25T15:21:07.276978+0000","last_peered":"2024-04-25T15:21:07.276978+0000","last_clean":"2024-04-25T15:21:07.276978+0000","last_became_active":"2024-04-25T15:21:07.270686+0000","last_became_peered":"2024-04-25T15:21:07.270686+0000","last_unstale":"2024-04-25T15:21:07.276978+0000","last_undegraded":"2024-04-25T15:21:07.276978+0000","last_fullsized":"2024-04-25T15:21:07.276978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:14.556209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253688+0000","last_change":"2024-04-25T15:21:07.253688+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253688+0000","last_undegraded":"2024-04-25T15:21:07.253688+0000","last_fullsized":"2024-04-25T15:21:07.253688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276948+0000","last_change":"2024-04-25T15:21:07.270720+0000","last_active":"2024-04-25T15:21:07.276948+0000","last_peered":"2024-04-25T15:21:07.276948+0000","last_clean":"2024-04-25T15:21:07.276948+0000","last_became_active":"2024-04-25T15:21:07.270418+0000","last_became_peered":"2024-04-25T15:21:07.270418+0000","last_unstale":"2024-04-25T15:21:07.276948+0000","last_undegraded":"2024-04-25T15:21:07.276948+0000","last_fullsized":"2024-04-25T15:21:07.276948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:23.664366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254153+0000","last_change":"2024-04-25T15:21:07.254153+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254153+0000","last_undegraded":"2024-04-25T15:21:07.254153+0000","last_fullsized":"2024-04-25T15:21:07.254153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274957+0000","last_change":"2024-04-25T15:21:07.270169+0000","last_active":"2024-04-25T15:21:07.274957+0000","last_peered":"2024-04-25T15:21:07.274957+0000","last_clean":"2024-04-25T15:21:07.274957+0000","last_became_active":"2024-04-25T15:21:07.269753+0000","last_became_peered":"2024-04-25T15:21:07.269753+0000","last_unstale":"2024-04-25T15:21:07.274957+0000","last_undegraded":"2024-04-25T15:21:07.274957+0000","last_fullsized":"2024-04-25T15:21:07.274957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:37.213221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257535+0000","last_change":"2024-04-25T15:21:07.257535+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257535+0000","last_undegraded":"2024-04-25T15:21:07.257535+0000","last_fullsized":"2024-04-25T15:21:07.257535+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272859+0000","last_change":"2024-04-25T15:21:07.269932+0000","last_active":"2024-04-25T15:21:07.272859+0000","last_peered":"2024-04-25T15:21:07.272859+0000","last_clean":"2024-04-25T15:21:07.272859+0000","last_became_active":"2024-04-25T15:21:07.269598+0000","last_became_peered":"2024-04-25T15:21:07.269598+0000","last_unstale":"2024-04-25T15:21:07.272859+0000","last_undegraded":"2024-04-25T15:21:07.272859+0000","last_fullsized":"2024-04-25T15:21:07.272859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:39.459934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256124+0000","last_change":"2024-04-25T15:21:07.256124+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256124+0000","last_undegraded":"2024-04-25T15:21:07.256124+0000","last_fullsized":"2024-04-25T15:21:07.256124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270365+0000","last_change":"2024-04-25T15:21:07.263451+0000","last_active":"2024-04-25T15:21:07.270365+0000","last_peered":"2024-04-25T15:21:07.270365+0000","last_clean":"2024-04-25T15:21:07.270365+0000","last_became_active":"2024-04-25T15:21:07.263111+0000","last_became_peered":"2024-04-25T15:21:07.263111+0000","last_unstale":"2024-04-25T15:21:07.270365+0000","last_undegraded":"2024-04-25T15:21:07.270365+0000","last_fullsized":"2024-04-25T15:21:07.270365+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261818+0000","last_change":"2024-04-25T15:21:07.261818+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261818+0000","last_undegraded":"2024-04-25T15:21:07.261818+0000","last_fullsized":"2024-04-25T15:21:07.261818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277011+0000","last_change":"2024-04-25T15:21:07.271421+0000","last_active":"2024-04-25T15:21:07.277011+0000","last_peered":"2024-04-25T15:21:07.277011+0000","last_clean":"2024-04-25T15:21:07.277011+0000","last_became_active":"2024-04-25T15:21:07.271136+0000","last_became_peered":"2024-04-25T15:21:07.271136+0000","last_unstale":"2024-04-25T15:21:07.277011+0000","last_undegraded":"2024-04-25T15:21:07.277011+0000","last_fullsized":"2024-04-25T15:21:07.277011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:34.931594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258775+0000","last_change":"2024-04-25T15:21:07.258775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258775+0000","last_undegraded":"2024-04-25T15:21:07.258775+0000","last_fullsized":"2024-04-25T15:21:07.258775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269969+0000","last_change":"2024-04-25T15:21:07.267201+0000","last_active":"2024-04-25T15:21:07.269969+0000","last_peered":"2024-04-25T15:21:07.269969+0000","last_clean":"2024-04-25T15:21:07.269969+0000","last_became_active":"2024-04-25T15:21:07.266901+0000","last_became_peered":"2024-04-25T15:21:07.266901+0000","last_unstale":"2024-04-25T15:21:07.269969+0000","last_undegraded":"2024-04-25T15:21:07.269969+0000","last_fullsized":"2024-04-25T15:21:07.269969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253682+0000","last_change":"2024-04-25T15:21:07.253682+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253682+0000","last_undegraded":"2024-04-25T15:21:07.253682+0000","last_fullsized":"2024-04-25T15:21:07.253682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270067+0000","last_change":"2024-04-25T15:21:07.265869+0000","last_active":"2024-04-25T15:21:07.270067+0000","last_peered":"2024-04-25T15:21:07.270067+0000","last_clean":"2024-04-25T15:21:07.270067+0000","last_became_active":"2024-04-25T15:21:07.265566+0000","last_became_peered":"2024-04-25T15:21:07.265566+0000","last_unstale":"2024-04-25T15:21:07.270067+0000","last_undegraded":"2024-04-25T15:21:07.270067+0000","last_fullsized":"2024-04-25T15:21:07.270067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256908+0000","last_change":"2024-04-25T15:21:07.256908+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256908+0000","last_undegraded":"2024-04-25T15:21:07.256908+0000","last_fullsized":"2024-04-25T15:21:07.256908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272141+0000","last_change":"2024-04-25T15:21:07.267132+0000","last_active":"2024-04-25T15:21:07.272141+0000","last_peered":"2024-04-25T15:21:07.272141+0000","last_clean":"2024-04-25T15:21:07.272141+0000","last_became_active":"2024-04-25T15:21:07.266783+0000","last_became_peered":"2024-04-25T15:21:07.266783+0000","last_unstale":"2024-04-25T15:21:07.272141+0000","last_undegraded":"2024-04-25T15:21:07.272141+0000","last_fullsized":"2024-04-25T15:21:07.272141+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:10.461067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255866+0000","last_change":"2024-04-25T15:21:07.255866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255866+0000","last_undegraded":"2024-04-25T15:21:07.255866+0000","last_fullsized":"2024-04-25T15:21:07.255866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272473+0000","last_change":"2024-04-25T15:21:07.268347+0000","last_active":"2024-04-25T15:21:07.272473+0000","last_peered":"2024-04-25T15:21:07.272473+0000","last_clean":"2024-04-25T15:21:07.272473+0000","last_became_active":"2024-04-25T15:21:07.268032+0000","last_became_peered":"2024-04-25T15:21:07.268032+0000","last_unstale":"2024-04-25T15:21:07.272473+0000","last_undegraded":"2024-04-25T15:21:07.272473+0000","last_fullsized":"2024-04-25T15:21:07.272473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260480+0000","last_change":"2024-04-25T15:21:07.260480+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260480+0000","last_undegraded":"2024-04-25T15:21:07.260480+0000","last_fullsized":"2024-04-25T15:21:07.260480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279438+0000","last_change":"2024-04-25T15:21:07.272594+0000","last_active":"2024-04-25T15:21:07.279438+0000","last_peered":"2024-04-25T15:21:07.279438+0000","last_clean":"2024-04-25T15:21:07.279438+0000","last_became_active":"2024-04-25T15:21:07.272274+0000","last_became_peered":"2024-04-25T15:21:07.272274+0000","last_unstale":"2024-04-25T15:21:07.279438+0000","last_undegraded":"2024-04-25T15:21:07.279438+0000","last_fullsized":"2024-04-25T15:21:07.279438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:09.551367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254935+0000","last_change":"2024-04-25T15:21:07.254935+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254935+0000","last_undegraded":"2024-04-25T15:21:07.254935+0000","last_fullsized":"2024-04-25T15:21:07.254935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274974+0000","last_change":"2024-04-25T15:21:07.271612+0000","last_active":"2024-04-25T15:21:07.274974+0000","last_peered":"2024-04-25T15:21:07.274974+0000","last_clean":"2024-04-25T15:21:07.274974+0000","last_became_active":"2024-04-25T15:21:07.271320+0000","last_became_peered":"2024-04-25T15:21:07.271320+0000","last_unstale":"2024-04-25T15:21:07.274974+0000","last_undegraded":"2024-04-25T15:21:07.274974+0000","last_fullsized":"2024-04-25T15:21:07.274974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:42.520351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":141,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.259258+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:21:07.259258+0000","last_peered":"2024-04-25T15:21:07.259258+0000","last_clean":"2024-04-25T15:21:07.259258+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:21:07.259258+0000","last_undegraded":"2024-04-25T15:21:07.259258+0000","last_fullsized":"2024-04-25T15:21:07.259258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1019904,"data_stored":773544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:21:13.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-04-25T15:21:13.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258465+0000', 'last_change': '2024-04-25T15:21:07.258465+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258465+0000', 'last_undegraded': '2024-04-25T15:21:07.258465+0000', 'last_fullsized': '2024-04-25T15:21:07.258465+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-04-25T15:21:13.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.259346+0000', 'last_change': '2024-04-25T15:21:07.259346+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.259346+0000', 'last_undegraded': '2024-04-25T15:21:07.259346+0000', 'last_fullsized': '2024-04-25T15:21:07.259346+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.265056+0000', 'last_change': '2024-04-25T15:21:07.265056+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.265056+0000', 'last_undegraded': '2024-04-25T15:21:07.265056+0000', 'last_fullsized': '2024-04-25T15:21:07.265056+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258425+0000', 'last_change': '2024-04-25T15:21:07.258425+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258425+0000', 'last_undegraded': '2024-04-25T15:21:07.258425+0000', 'last_fullsized': '2024-04-25T15:21:07.258425+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.260423+0000', 'last_change': '2024-04-25T15:21:07.260423+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.260423+0000', 'last_undegraded': '2024-04-25T15:21:07.260423+0000', 'last_fullsized': '2024-04-25T15:21:07.260423+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256056+0000', 'last_change': '2024-04-25T15:21:07.256056+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256056+0000', 'last_undegraded': '2024-04-25T15:21:07.256056+0000', 'last_fullsized': '2024-04-25T15:21:07.256056+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-04-25T15:21:13.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.253502+0000', 'last_change': '2024-04-25T15:21:07.253502+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.253502+0000', 'last_undegraded': '2024-04-25T15:21:07.253502+0000', 'last_fullsized': '2024-04-25T15:21:07.253502+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.259316+0000', 'last_change': '2024-04-25T15:21:07.259316+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.259316+0000', 'last_undegraded': '2024-04-25T15:21:07.259316+0000', 'last_fullsized': '2024-04-25T15:21:07.259316+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.264325+0000', 'last_change': '2024-04-25T15:21:07.264325+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.264325+0000', 'last_undegraded': '2024-04-25T15:21:07.264325+0000', 'last_fullsized': '2024-04-25T15:21:07.264325+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.252907+0000', 'last_change': '2024-04-25T15:21:07.252907+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.252907+0000', 'last_undegraded': '2024-04-25T15:21:07.252907+0000', 'last_fullsized': '2024-04-25T15:21:07.252907+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256157+0000', 'last_change': '2024-04-25T15:21:07.256157+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256157+0000', 'last_undegraded': '2024-04-25T15:21:07.256157+0000', 'last_fullsized': '2024-04-25T15:21:07.256157+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.255957+0000', 'last_change': '2024-04-25T15:21:07.255957+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.255957+0000', 'last_undegraded': '2024-04-25T15:21:07.255957+0000', 'last_fullsized': '2024-04-25T15:21:07.255957+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.255463+0000', 'last_change': '2024-04-25T15:21:07.255463+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.255463+0000', 'last_undegraded': '2024-04-25T15:21:07.255463+0000', 'last_fullsized': '2024-04-25T15:21:07.255463+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257032+0000', 'last_change': '2024-04-25T15:21:07.257032+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257032+0000', 'last_undegraded': '2024-04-25T15:21:07.257032+0000', 'last_fullsized': '2024-04-25T15:21:07.257032+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.252787+0000', 'last_change': '2024-04-25T15:21:07.252787+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.252787+0000', 'last_undegraded': '2024-04-25T15:21:07.252787+0000', 'last_fullsized': '2024-04-25T15:21:07.252787+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258605+0000', 'last_change': '2024-04-25T15:21:07.258605+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258605+0000', 'last_undegraded': '2024-04-25T15:21:07.258605+0000', 'last_fullsized': '2024-04-25T15:21:07.258605+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.260477+0000', 'last_change': '2024-04-25T15:21:07.260477+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.260477+0000', 'last_undegraded': '2024-04-25T15:21:07.260477+0000', 'last_fullsized': '2024-04-25T15:21:07.260477+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-04-25T15:21:13.004 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.260775+0000', 'last_change': '2024-04-25T15:21:07.260775+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.260775+0000', 'last_undegraded': '2024-04-25T15:21:07.260775+0000', 'last_fullsized': '2024-04-25T15:21:07.260775+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.261610+0000', 'last_change': '2024-04-25T15:21:07.261610+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.261610+0000', 'last_undegraded': '2024-04-25T15:21:07.261610+0000', 'last_fullsized': '2024-04-25T15:21:07.261610+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.260690+0000', 'last_change': '2024-04-25T15:21:07.260690+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.260690+0000', 'last_undegraded': '2024-04-25T15:21:07.260690+0000', 'last_fullsized': '2024-04-25T15:21:07.260690+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254693+0000', 'last_change': '2024-04-25T15:21:07.254693+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254693+0000', 'last_undegraded': '2024-04-25T15:21:07.254693+0000', 'last_fullsized': '2024-04-25T15:21:07.254693+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.255362+0000', 'last_change': '2024-04-25T15:21:07.255362+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.255362+0000', 'last_undegraded': '2024-04-25T15:21:07.255362+0000', 'last_fullsized': '2024-04-25T15:21:07.255362+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257566+0000', 'last_change': '2024-04-25T15:21:07.257566+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257566+0000', 'last_undegraded': '2024-04-25T15:21:07.257566+0000', 'last_fullsized': '2024-04-25T15:21:07.257566+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-04-25T15:21:13.005 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257299+0000', 'last_change': '2024-04-25T15:21:07.257299+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257299+0000', 'last_undegraded': '2024-04-25T15:21:07.257299+0000', 'last_fullsized': '2024-04-25T15:21:07.257299+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.261464+0000', 'last_change': '2024-04-25T15:21:07.261464+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.261464+0000', 'last_undegraded': '2024-04-25T15:21:07.261464+0000', 'last_fullsized': '2024-04-25T15:21:07.261464+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256914+0000', 'last_change': '2024-04-25T15:21:07.256914+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256914+0000', 'last_undegraded': '2024-04-25T15:21:07.256914+0000', 'last_fullsized': '2024-04-25T15:21:07.256914+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.259855+0000', 'last_change': '2024-04-25T15:21:07.259855+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.259855+0000', 'last_undegraded': '2024-04-25T15:21:07.259855+0000', 'last_fullsized': '2024-04-25T15:21:07.259855+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.252242+0000', 'last_change': '2024-04-25T15:21:07.252242+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.252242+0000', 'last_undegraded': '2024-04-25T15:21:07.252242+0000', 'last_fullsized': '2024-04-25T15:21:07.252242+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.260409+0000', 'last_change': '2024-04-25T15:21:07.260409+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.260409+0000', 'last_undegraded': '2024-04-25T15:21:07.260409+0000', 'last_fullsized': '2024-04-25T15:21:07.260409+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:21:13.006 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.259146+0000', 'last_change': '2024-04-25T15:21:07.259146+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.259146+0000', 'last_undegraded': '2024-04-25T15:21:07.259146+0000', 'last_fullsized': '2024-04-25T15:21:07.259146+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.255377+0000', 'last_change': '2024-04-25T15:21:07.255377+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.255377+0000', 'last_undegraded': '2024-04-25T15:21:07.255377+0000', 'last_fullsized': '2024-04-25T15:21:07.255377+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254666+0000', 'last_change': '2024-04-25T15:21:07.254666+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254666+0000', 'last_undegraded': '2024-04-25T15:21:07.254666+0000', 'last_fullsized': '2024-04-25T15:21:07.254666+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257141+0000', 'last_change': '2024-04-25T15:21:07.257141+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257141+0000', 'last_undegraded': '2024-04-25T15:21:07.257141+0000', 'last_fullsized': '2024-04-25T15:21:07.257141+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-04-25T15:21:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257143+0000', 'last_change': '2024-04-25T15:21:07.257143+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257143+0000', 'last_undegraded': '2024-04-25T15:21:07.257143+0000', 'last_fullsized': '2024-04-25T15:21:07.257143+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254113+0000', 'last_change': '2024-04-25T15:21:07.254113+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254113+0000', 'last_undegraded': '2024-04-25T15:21:07.254113+0000', 'last_fullsized': '2024-04-25T15:21:07.254113+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258984+0000', 'last_change': '2024-04-25T15:21:07.258984+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258984+0000', 'last_undegraded': '2024-04-25T15:21:07.258984+0000', 'last_fullsized': '2024-04-25T15:21:07.258984+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.255378+0000', 'last_change': '2024-04-25T15:21:07.255378+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.255378+0000', 'last_undegraded': '2024-04-25T15:21:07.255378+0000', 'last_fullsized': '2024-04-25T15:21:07.255378+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.252100+0000', 'last_change': '2024-04-25T15:21:07.252100+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.252100+0000', 'last_undegraded': '2024-04-25T15:21:07.252100+0000', 'last_fullsized': '2024-04-25T15:21:07.252100+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.253544+0000', 'last_change': '2024-04-25T15:21:07.253544+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.253544+0000', 'last_undegraded': '2024-04-25T15:21:07.253544+0000', 'last_fullsized': '2024-04-25T15:21:07.253544+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.008 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258866+0000', 'last_change': '2024-04-25T15:21:07.258866+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258866+0000', 'last_undegraded': '2024-04-25T15:21:07.258866+0000', 'last_fullsized': '2024-04-25T15:21:07.258866+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257928+0000', 'last_change': '2024-04-25T15:21:07.257928+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257928+0000', 'last_undegraded': '2024-04-25T15:21:07.257928+0000', 'last_fullsized': '2024-04-25T15:21:07.257928+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.262262+0000', 'last_change': '2024-04-25T15:21:07.262262+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.262262+0000', 'last_undegraded': '2024-04-25T15:21:07.262262+0000', 'last_fullsized': '2024-04-25T15:21:07.262262+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254905+0000', 'last_change': '2024-04-25T15:21:07.254905+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254905+0000', 'last_undegraded': '2024-04-25T15:21:07.254905+0000', 'last_fullsized': '2024-04-25T15:21:07.254905+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.252968+0000', 'last_change': '2024-04-25T15:21:07.252968+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.252968+0000', 'last_undegraded': '2024-04-25T15:21:07.252968+0000', 'last_fullsized': '2024-04-25T15:21:07.252968+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-04-25T15:21:13.009 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254326+0000', 'last_change': '2024-04-25T15:21:07.254326+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254326+0000', 'last_undegraded': '2024-04-25T15:21:07.254326+0000', 'last_fullsized': '2024-04-25T15:21:07.254326+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257171+0000', 'last_change': '2024-04-25T15:21:07.257171+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257171+0000', 'last_undegraded': '2024-04-25T15:21:07.257171+0000', 'last_fullsized': '2024-04-25T15:21:07.257171+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256033+0000', 'last_change': '2024-04-25T15:21:07.256033+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256033+0000', 'last_undegraded': '2024-04-25T15:21:07.256033+0000', 'last_fullsized': '2024-04-25T15:21:07.256033+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256795+0000', 'last_change': '2024-04-25T15:21:07.256795+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256795+0000', 'last_undegraded': '2024-04-25T15:21:07.256795+0000', 'last_fullsized': '2024-04-25T15:21:07.256795+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.259420+0000', 'last_change': '2024-04-25T15:21:07.259420+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.259420+0000', 'last_undegraded': '2024-04-25T15:21:07.259420+0000', 'last_fullsized': '2024-04-25T15:21:07.259420+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254229+0000', 'last_change': '2024-04-25T15:21:07.254229+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254229+0000', 'last_undegraded': '2024-04-25T15:21:07.254229+0000', 'last_fullsized': '2024-04-25T15:21:07.254229+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.010 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256308+0000', 'last_change': '2024-04-25T15:21:07.256308+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256308+0000', 'last_undegraded': '2024-04-25T15:21:07.256308+0000', 'last_fullsized': '2024-04-25T15:21:07.256308+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258937+0000', 'last_change': '2024-04-25T15:21:07.258937+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258937+0000', 'last_undegraded': '2024-04-25T15:21:07.258937+0000', 'last_fullsized': '2024-04-25T15:21:07.258937+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256563+0000', 'last_change': '2024-04-25T15:21:07.256563+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256563+0000', 'last_undegraded': '2024-04-25T15:21:07.256563+0000', 'last_fullsized': '2024-04-25T15:21:07.256563+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.253688+0000', 'last_change': '2024-04-25T15:21:07.253688+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.253688+0000', 'last_undegraded': '2024-04-25T15:21:07.253688+0000', 'last_fullsized': '2024-04-25T15:21:07.253688+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254153+0000', 'last_change': '2024-04-25T15:21:07.254153+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254153+0000', 'last_undegraded': '2024-04-25T15:21:07.254153+0000', 'last_fullsized': '2024-04-25T15:21:07.254153+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-04-25T15:21:13.011 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.257535+0000', 'last_change': '2024-04-25T15:21:07.257535+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.257535+0000', 'last_undegraded': '2024-04-25T15:21:07.257535+0000', 'last_fullsized': '2024-04-25T15:21:07.257535+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256124+0000', 'last_change': '2024-04-25T15:21:07.256124+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256124+0000', 'last_undegraded': '2024-04-25T15:21:07.256124+0000', 'last_fullsized': '2024-04-25T15:21:07.256124+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.261818+0000', 'last_change': '2024-04-25T15:21:07.261818+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.261818+0000', 'last_undegraded': '2024-04-25T15:21:07.261818+0000', 'last_fullsized': '2024-04-25T15:21:07.261818+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.258775+0000', 'last_change': '2024-04-25T15:21:07.258775+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.258775+0000', 'last_undegraded': '2024-04-25T15:21:07.258775+0000', 'last_fullsized': '2024-04-25T15:21:07.258775+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.253682+0000', 'last_change': '2024-04-25T15:21:07.253682+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.253682+0000', 'last_undegraded': '2024-04-25T15:21:07.253682+0000', 'last_fullsized': '2024-04-25T15:21:07.253682+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.256908+0000', 'last_change': '2024-04-25T15:21:07.256908+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.256908+0000', 'last_undegraded': '2024-04-25T15:21:07.256908+0000', 'last_fullsized': '2024-04-25T15:21:07.256908+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:21:13.012 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-04-25T15:21:13.013 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.255866+0000', 'last_change': '2024-04-25T15:21:07.255866+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.255866+0000', 'last_undegraded': '2024-04-25T15:21:07.255866+0000', 'last_fullsized': '2024-04-25T15:21:07.255866+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:21:13.013 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-25T15:21:13.013 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.260480+0000', 'last_change': '2024-04-25T15:21:07.260480+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.260480+0000', 'last_undegraded': '2024-04-25T15:21:07.260480+0000', 'last_fullsized': '2024-04-25T15:21:07.260480+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:21:13.013 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-04-25T15:21:13.013 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:21:07.254935+0000', 'last_change': '2024-04-25T15:21:07.254935+0000', 'last_active': '2024-04-25T15:21:07.245862+0000', 'last_peered': '2024-04-25T15:21:07.245862+0000', 'last_clean': '2024-04-25T15:21:07.245862+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:21:07.254935+0000', 'last_undegraded': '2024-04-25T15:21:07.254935+0000', 'last_fullsized': '2024-04-25T15:21:07.254935+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'last_clean_scrub_stamp': '2024-04-25T15:21:07.245862+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:21:13.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T15:21:13.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.144+0000 7ffaa9015640 1 Processor -- start 2024-04-25T15:21:13.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.144+0000 7ffaa9015640 1 -- start start 2024-04-25T15:21:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa9015640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffaa40d4f60 0x7ffaa40d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa9015640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffaa406ccc0 0x7ffaa406d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa9015640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa40b9200 con 0x7ffaa40d0020 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa9015640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa406c550 con 0x7ffaa406ccc0 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa9015640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa406c830 con 0x7ffaa40d4f60 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa359e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffaa406ccc0 0x7ffaa406d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa359e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffaa406ccc0 0x7ffaa406d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38132/0 (socket says 172.21.15.146:38132) 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa359e640 1 -- 172.21.15.146:0/3211038557 learned_addr learned my addr 172.21.15.146:0/3211038557 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.145+0000 7ffaa2d9d640 1 --2- 172.21.15.146:0/3211038557 >> v2:172.21.15.177:3300/0 conn(0x7ffaa40d4f60 0x7ffaa40d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa359e640 1 -- 172.21.15.146:0/3211038557 >> v2:172.21.15.177:3300/0 conn(0x7ffaa40d4f60 msgr2=0x7ffaa40d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa359e640 1 --2- 172.21.15.146:0/3211038557 >> v2:172.21.15.177:3300/0 conn(0x7ffaa40d4f60 0x7ffaa40d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa359e640 1 -- 172.21.15.146:0/3211038557 >> v1:172.21.15.177:6789/0 conn(0x7ffaa40d0020 legacy=0x7ffaa40d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa359e640 1 -- 172.21.15.146:0/3211038557 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa40d1070 con 0x7ffaa406ccc0 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa359e640 1 --2- 172.21.15.146:0/3211038557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa406ccc0 0x7ffaa406d0d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40d5930 tx=0x7ffa9c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23b456199a65ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa1d9b640 1 -- 172.21.15.146:0/3211038557 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c036070 con 0x7ffaa406ccc0 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa1d9b640 1 -- 172.21.15.146:0/3211038557 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9c031600 con 0x7ffaa406ccc0 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.146+0000 7ffaa1d9b640 1 -- 172.21.15.146:0/3211038557 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c0319d0 con 0x7ffaa406ccc0 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 -- 172.21.15.146:0/3211038557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa406ccc0 msgr2=0x7ffaa406d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 --2- 172.21.15.146:0/3211038557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa406ccc0 0x7ffaa406d0d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40d5930 tx=0x7ffa9c02a610 comp rx=0 tx=0).stop 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 -- 172.21.15.146:0/3211038557 shutdown_connections 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 --2- 172.21.15.146:0/3211038557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa406ccc0 0x7ffaa406d0d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 --2- 172.21.15.146:0/3211038557 >> v2:172.21.15.177:3300/0 conn(0x7ffaa40d4f60 0x7ffaa40d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 -- 172.21.15.146:0/3211038557 >> 172.21.15.146:0/3211038557 conn(0x7ffaa40d2450 msgr2=0x7ffaa40d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 -- 172.21.15.146:0/3211038557 shutdown_connections 2024-04-25T15:21:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.147+0000 7ffaa9015640 1 -- 172.21.15.146:0/3211038557 wait complete. 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.149+0000 7ffaa9015640 1 Processor -- start 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.149+0000 7ffaa9015640 1 -- start start 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.150+0000 7ffaa9015640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaa406ccc0 0x7ffaa41f9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa9015640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaa40d0020 0x7ffaa4158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa9015640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 0x7ffaa41ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa9015640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa40b5150 con 0x7ffaa40d4f60 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa9015640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa40d15f0 con 0x7ffaa40d0020 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa9015640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa40b52d0 con 0x7ffaa406ccc0 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 0x7ffaa41ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 0x7ffaa41ebd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38134/0 (socket says 172.21.15.146:38134) 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 -- 172.21.15.146:0/4240658240 learned_addr learned my addr 172.21.15.146:0/4240658240 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa2d9d640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaa406ccc0 0x7ffaa41f9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa259c640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaa40d0020 0x7ffaa4158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 -- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaa406ccc0 msgr2=0x7ffaa41f9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaa406ccc0 0x7ffaa41f9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 -- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaa40d0020 msgr2=0x7ffaa4158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaa40d0020 0x7ffaa4158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 -- 172.21.15.146:0/4240658240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa4158bb0 con 0x7ffaa40d4f60 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa359e640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 0x7ffaa41ebd30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c02ac00 tx=0x7ffa9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:13.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c03f020 con 0x7ffaa40d4f60 2024-04-25T15:21:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9c037040 con 0x7ffaa40d4f60 2024-04-25T15:21:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa41610a0 con 0x7ffaa40d4f60 2024-04-25T15:21:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.151+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c036070 con 0x7ffaa40d4f60 2024-04-25T15:21:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.152+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa4161620 con 0x7ffaa40d4f60 2024-04-25T15:21:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.152+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa406a0e0 con 0x7ffaa40d4f60 2024-04-25T15:21:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.153+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffa9c042020 con 0x7ffaa40d4f60 2024-04-25T15:21:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.154+0000 7ffa8b7fe640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa900744b0 0x7ffa900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.154+0000 7ffaa2d9d640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa900744b0 0x7ffa900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.154+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ffa9c0b5a70 con 0x7ffaa40d4f60 2024-04-25T15:21:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.154+0000 7ffaa2d9d640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa900744b0 0x7ffa900769c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40b4880 tx=0x7ffa94009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.158+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa9c049020 con 0x7ffaa40d4f60 2024-04-25T15:21:13.246 INFO:tasks.ceph.mds.b.smithi177.stderr:2024-04-25T15:21:13.245+0000 7fb046c62640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:21:13.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.403+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7ffaa40b5d90 con 0x7ffaa40d4f60 2024-04-25T15:21:13.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.403+0000 7ffa8b7fe640 1 -- 172.21.15.146:0/4240658240 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1034 (secure 0 0 0) 0x7ffaa41610a0 con 0x7ffaa40d4f60 2024-04-25T15:21:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:13.404 INFO:teuthology.orchestra.run.smithi146.stdout:{"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":111,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1714058372,"num_in_osds":8,"osd_in_since":1714058364,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225939456,"bytes_avail":772868173824,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"b","status":"up:active","gid":5556}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-25T15:20:52.001838+0000","services":{}},"progress_events":{}} 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa900744b0 msgr2=0x7ffa900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa900744b0 0x7ffa900769c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40b4880 tx=0x7ffa94009190 comp rx=0 tx=0).stop 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 msgr2=0x7ffaa41ebd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 0x7ffaa41ebd30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c02ac00 tx=0x7ffa9c004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 shutdown_connections 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa900744b0 0x7ffa900769c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaa40d4f60 0x7ffaa41ebd30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaa40d0020 0x7ffaa4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 --2- 172.21.15.146:0/4240658240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffaa406ccc0 0x7ffaa41f9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 >> 172.21.15.146:0/4240658240 conn(0x7ffaa40d2450 msgr2=0x7ffaa40d4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:13.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 shutdown_connections 2024-04-25T15:21:13.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.405+0000 7ffaa9015640 1 -- 172.21.15.146:0/4240658240 wait complete. 2024-04-25T15:21:13.417 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T15:21:13.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:21:13.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.524+0000 7fa24f38f640 1 Processor -- start 2024-04-25T15:21:13.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.524+0000 7fa24f38f640 1 -- start start 2024-04-25T15:21:13.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24f38f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa2480cfbe0 0x7fa2480cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24f38f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa2480d19d0 0x7fa2480d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24f38f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480b3920 0x7fa2480b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24f38f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa2480b5020 con 0x7fa2480cfbe0 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24f38f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa2480b4a60 con 0x7fa2480d19d0 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24f38f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa2480b4d40 con 0x7fa2480b3920 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24d12b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa2480cfbe0 0x7fa2480cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24d12b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa2480cfbe0 0x7fa2480cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38150/0 (socket says 172.21.15.146:38150) 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.525+0000 7fa24d12b640 1 -- 172.21.15.146:0/2413524517 learned_addr learned my addr 172.21.15.146:0/2413524517 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24c92a640 1 --2- 172.21.15.146:0/2413524517 >> v2:172.21.15.177:3300/0 conn(0x7fa2480d19d0 0x7fa2480d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d92c640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480b3920 0x7fa2480b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d12b640 1 -- 172.21.15.146:0/2413524517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480b3920 msgr2=0x7fa2480b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d12b640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480b3920 0x7fa2480b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d12b640 1 -- 172.21.15.146:0/2413524517 >> v2:172.21.15.177:3300/0 conn(0x7fa2480d19d0 msgr2=0x7fa2480d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d12b640 1 --2- 172.21.15.146:0/2413524517 >> v2:172.21.15.177:3300/0 conn(0x7fa2480d19d0 0x7fa2480d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d12b640 1 -- 172.21.15.146:0/2413524517 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2480cead0 con 0x7fa2480cfbe0 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24c92a640 1 --2- 172.21.15.146:0/2413524517 >> v2:172.21.15.177:3300/0 conn(0x7fa2480d19d0 0x7fa2480d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d92c640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480b3920 0x7fa2480b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:13.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.526+0000 7fa24d12b640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480cfbe0 0x7fa2480cf260 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa2480b8510 tx=0x7fa24402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f87e56447d3b37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:13.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa23ffff640 1 -- 172.21.15.146:0/2413524517 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244004030 con 0x7fa2480cfbe0 2024-04-25T15:21:13.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa23ffff640 1 -- 172.21.15.146:0/2413524517 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2440041d0 con 0x7fa2480cfbe0 2024-04-25T15:21:13.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa23ffff640 1 -- 172.21.15.146:0/2413524517 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244003610 con 0x7fa2480cfbe0 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 -- 172.21.15.146:0/2413524517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480cfbe0 msgr2=0x7fa2480cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480cfbe0 0x7fa2480cf260 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa2480b8510 tx=0x7fa24402a250 comp rx=0 tx=0).stop 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 -- 172.21.15.146:0/2413524517 shutdown_connections 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480b3920 0x7fa2480b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 --2- 172.21.15.146:0/2413524517 >> v2:172.21.15.177:3300/0 conn(0x7fa2480d19d0 0x7fa2480d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 --2- 172.21.15.146:0/2413524517 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480cfbe0 0x7fa2480cf260 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 -- 172.21.15.146:0/2413524517 >> 172.21.15.146:0/2413524517 conn(0x7fa2480b73b0 msgr2=0x7fa2480b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 -- 172.21.15.146:0/2413524517 shutdown_connections 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.527+0000 7fa24f38f640 1 -- 172.21.15.146:0/2413524517 wait complete. 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.529+0000 7fa24f38f640 1 Processor -- start 2024-04-25T15:21:13.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.529+0000 7fa24f38f640 1 -- start start 2024-04-25T15:21:13.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.530+0000 7fa24f38f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2480b3920 0x7fa24809d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.530+0000 7fa24f38f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480cfbe0 0x7fa248071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.530+0000 7fa24f38f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 0x7fa248071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.530+0000 7fa24f38f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa2480cf7a0 con 0x7fa2480d19d0 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.530+0000 7fa24f38f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa2480d2320 con 0x7fa2480b3920 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.530+0000 7fa24f38f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa2480c82d0 con 0x7fa2480cfbe0 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 0x7fa248071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 0x7fa248071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38166/0 (socket says 172.21.15.146:38166) 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 -- 172.21.15.146:0/1535284249 learned_addr learned my addr 172.21.15.146:0/1535284249 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24c92a640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480cfbe0 0x7fa248071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d12b640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2480b3920 0x7fa24809d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 -- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480cfbe0 msgr2=0x7fa248071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480cfbe0 0x7fa248071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 -- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2480b3920 msgr2=0x7fa24809d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2480b3920 0x7fa24809d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 -- 172.21.15.146:0/1535284249 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2481f6700 con 0x7fa2480d19d0 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24c92a640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480cfbe0 0x7fa248071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d12b640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2480b3920 0x7fa24809d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24d92c640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 0x7fa248071ab0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa24000cd10 tx=0x7fa24000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000f680 con 0x7fa2480d19d0 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa24000f840 con 0x7fa2480d19d0 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2481f69e0 con 0x7fa2480d19d0 2024-04-25T15:21:13.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24000fc30 con 0x7fa2480d19d0 2024-04-25T15:21:13.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.531+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa24809f4f0 con 0x7fa2480d19d0 2024-04-25T15:21:13.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.533+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa24806a0e0 con 0x7fa2480d19d0 2024-04-25T15:21:13.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.533+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa2400213f0 con 0x7fa2480d19d0 2024-04-25T15:21:13.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.533+0000 7fa23dffb640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa22c0744b0 0x7fa22c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:13.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.533+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa240096170 con 0x7fa2480d19d0 2024-04-25T15:21:13.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.533+0000 7fa24d12b640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa22c0744b0 0x7fa22c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:13.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.534+0000 7fa24d12b640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa22c0744b0 0x7fa22c0769c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa244033000 tx=0x7fa24402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:13.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.538+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa240014030 con 0x7fa2480d19d0 2024-04-25T15:21:13.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.708+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa2480cf920 con 0x7fa22c0744b0 2024-04-25T15:21:13.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.714+0000 7fa23dffb640 1 -- 172.21.15.146:0/1535284249 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246623 (secure 0 0 0) 0x7fa2480cf920 con 0x7fa22c0744b0 2024-04-25T15:21:13.717 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:13.718 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa22c0744b0 msgr2=0x7fa22c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa22c0744b0 0x7fa22c0769c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa244033000 tx=0x7fa24402e000 comp rx=0 tx=0).stop 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 msgr2=0x7fa248071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 0x7fa248071ab0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa24000cd10 tx=0x7fa24000a650 comp rx=0 tx=0).stop 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 shutdown_connections 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa22c0744b0 0x7fa22c0769c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2480d19d0 0x7fa248071ab0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2480cfbe0 0x7fa248071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 --2- 172.21.15.146:0/1535284249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2480b3920 0x7fa24809d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:13.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 >> 172.21.15.146:0/1535284249 conn(0x7fa2480b73b0 msgr2=0x7fa2480b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:13.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 shutdown_connections 2024-04-25T15:21:13.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:13.717+0000 7fa24f38f640 1 -- 172.21.15.146:0/1535284249 wait complete. 2024-04-25T15:21:13.729 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-25T15:21:12.005640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220644,"kb_used_data":4464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754076,"statfs":{"total":773094113280,"available":772868173824,"internally_reserved":0,"allocated":4571136,"data_stored":2632810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758904"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258465+0000","last_change":"2024-04-25T15:21:07.258465+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258465+0000","last_undegraded":"2024-04-25T15:21:07.258465+0000","last_fullsized":"2024-04-25T15:21:07.258465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259346+0000","last_change":"2024-04-25T15:21:07.259346+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259346+0000","last_undegraded":"2024-04-25T15:21:07.259346+0000","last_fullsized":"2024-04-25T15:21:07.259346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.265056+0000","last_change":"2024-04-25T15:21:07.265056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.265056+0000","last_undegraded":"2024-04-25T15:21:07.265056+0000","last_fullsized":"2024-04-25T15:21:07.265056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258425+0000","last_change":"2024-04-25T15:21:07.258425+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258425+0000","last_undegraded":"2024-04-25T15:21:07.258425+0000","last_fullsized":"2024-04-25T15:21:07.258425+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260423+0000","last_change":"2024-04-25T15:21:07.260423+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260423+0000","last_undegraded":"2024-04-25T15:21:07.260423+0000","last_fullsized":"2024-04-25T15:21:07.260423+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256056+0000","last_change":"2024-04-25T15:21:07.256056+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256056+0000","last_undegraded":"2024-04-25T15:21:07.256056+0000","last_fullsized":"2024-04-25T15:21:07.256056+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253502+0000","last_change":"2024-04-25T15:21:07.253502+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253502+0000","last_undegraded":"2024-04-25T15:21:07.253502+0000","last_fullsized":"2024-04-25T15:21:07.253502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259316+0000","last_change":"2024-04-25T15:21:07.259316+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259316+0000","last_undegraded":"2024-04-25T15:21:07.259316+0000","last_fullsized":"2024-04-25T15:21:07.259316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.264325+0000","last_change":"2024-04-25T15:21:07.264325+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.264325+0000","last_undegraded":"2024-04-25T15:21:07.264325+0000","last_fullsized":"2024-04-25T15:21:07.264325+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252907+0000","last_change":"2024-04-25T15:21:07.252907+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252907+0000","last_undegraded":"2024-04-25T15:21:07.252907+0000","last_fullsized":"2024-04-25T15:21:07.252907+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256157+0000","last_change":"2024-04-25T15:21:07.256157+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256157+0000","last_undegraded":"2024-04-25T15:21:07.256157+0000","last_fullsized":"2024-04-25T15:21:07.256157+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255957+0000","last_change":"2024-04-25T15:21:07.255957+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255957+0000","last_undegraded":"2024-04-25T15:21:07.255957+0000","last_fullsized":"2024-04-25T15:21:07.255957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255463+0000","last_change":"2024-04-25T15:21:07.255463+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255463+0000","last_undegraded":"2024-04-25T15:21:07.255463+0000","last_fullsized":"2024-04-25T15:21:07.255463+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257032+0000","last_change":"2024-04-25T15:21:07.257032+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257032+0000","last_undegraded":"2024-04-25T15:21:07.257032+0000","last_fullsized":"2024-04-25T15:21:07.257032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252787+0000","last_change":"2024-04-25T15:21:07.252787+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252787+0000","last_undegraded":"2024-04-25T15:21:07.252787+0000","last_fullsized":"2024-04-25T15:21:07.252787+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258605+0000","last_change":"2024-04-25T15:21:07.258605+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258605+0000","last_undegraded":"2024-04-25T15:21:07.258605+0000","last_fullsized":"2024-04-25T15:21:07.258605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260477+0000","last_change":"2024-04-25T15:21:07.260477+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260477+0000","last_undegraded":"2024-04-25T15:21:07.260477+0000","last_fullsized":"2024-04-25T15:21:07.260477+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260775+0000","last_change":"2024-04-25T15:21:07.260775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260775+0000","last_undegraded":"2024-04-25T15:21:07.260775+0000","last_fullsized":"2024-04-25T15:21:07.260775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261610+0000","last_change":"2024-04-25T15:21:07.261610+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261610+0000","last_undegraded":"2024-04-25T15:21:07.261610+0000","last_fullsized":"2024-04-25T15:21:07.261610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260690+0000","last_change":"2024-04-25T15:21:07.260690+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260690+0000","last_undegraded":"2024-04-25T15:21:07.260690+0000","last_fullsized":"2024-04-25T15:21:07.260690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254693+0000","last_change":"2024-04-25T15:21:07.254693+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254693+0000","last_undegraded":"2024-04-25T15:21:07.254693+0000","last_fullsized":"2024-04-25T15:21:07.254693+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255362+0000","last_change":"2024-04-25T15:21:07.255362+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255362+0000","last_undegraded":"2024-04-25T15:21:07.255362+0000","last_fullsized":"2024-04-25T15:21:07.255362+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257566+0000","last_change":"2024-04-25T15:21:07.257566+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257566+0000","last_undegraded":"2024-04-25T15:21:07.257566+0000","last_fullsized":"2024-04-25T15:21:07.257566+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257299+0000","last_change":"2024-04-25T15:21:07.257299+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257299+0000","last_undegraded":"2024-04-25T15:21:07.257299+0000","last_fullsized":"2024-04-25T15:21:07.257299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261464+0000","last_change":"2024-04-25T15:21:07.261464+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261464+0000","last_undegraded":"2024-04-25T15:21:07.261464+0000","last_fullsized":"2024-04-25T15:21:07.261464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256914+0000","last_change":"2024-04-25T15:21:07.256914+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256914+0000","last_undegraded":"2024-04-25T15:21:07.256914+0000","last_fullsized":"2024-04-25T15:21:07.256914+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259855+0000","last_change":"2024-04-25T15:21:07.259855+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259855+0000","last_undegraded":"2024-04-25T15:21:07.259855+0000","last_fullsized":"2024-04-25T15:21:07.259855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252242+0000","last_change":"2024-04-25T15:21:07.252242+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252242+0000","last_undegraded":"2024-04-25T15:21:07.252242+0000","last_fullsized":"2024-04-25T15:21:07.252242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260409+0000","last_change":"2024-04-25T15:21:07.260409+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260409+0000","last_undegraded":"2024-04-25T15:21:07.260409+0000","last_fullsized":"2024-04-25T15:21:07.260409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259146+0000","last_change":"2024-04-25T15:21:07.259146+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259146+0000","last_undegraded":"2024-04-25T15:21:07.259146+0000","last_fullsized":"2024-04-25T15:21:07.259146+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255377+0000","last_change":"2024-04-25T15:21:07.255377+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255377+0000","last_undegraded":"2024-04-25T15:21:07.255377+0000","last_fullsized":"2024-04-25T15:21:07.255377+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254666+0000","last_change":"2024-04-25T15:21:07.254666+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254666+0000","last_undegraded":"2024-04-25T15:21:07.254666+0000","last_fullsized":"2024-04-25T15:21:07.254666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257141+0000","last_change":"2024-04-25T15:21:07.257141+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257141+0000","last_undegraded":"2024-04-25T15:21:07.257141+0000","last_fullsized":"2024-04-25T15:21:07.257141+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277551+0000","last_change":"2024-04-25T15:21:07.273191+0000","last_active":"2024-04-25T15:21:07.277551+0000","last_peered":"2024-04-25T15:21:07.277551+0000","last_clean":"2024-04-25T15:21:07.277551+0000","last_became_active":"2024-04-25T15:21:07.272854+0000","last_became_peered":"2024-04-25T15:21:07.272854+0000","last_unstale":"2024-04-25T15:21:07.277551+0000","last_undegraded":"2024-04-25T15:21:07.277551+0000","last_fullsized":"2024-04-25T15:21:07.277551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:00.687305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257143+0000","last_change":"2024-04-25T15:21:07.257143+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257143+0000","last_undegraded":"2024-04-25T15:21:07.257143+0000","last_fullsized":"2024-04-25T15:21:07.257143+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270022+0000","last_change":"2024-04-25T15:21:07.268440+0000","last_active":"2024-04-25T15:21:07.270022+0000","last_peered":"2024-04-25T15:21:07.270022+0000","last_clean":"2024-04-25T15:21:07.270022+0000","last_became_active":"2024-04-25T15:21:07.268125+0000","last_became_peered":"2024-04-25T15:21:07.268125+0000","last_unstale":"2024-04-25T15:21:07.270022+0000","last_undegraded":"2024-04-25T15:21:07.270022+0000","last_fullsized":"2024-04-25T15:21:07.270022+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254113+0000","last_change":"2024-04-25T15:21:07.254113+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254113+0000","last_undegraded":"2024-04-25T15:21:07.254113+0000","last_fullsized":"2024-04-25T15:21:07.254113+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272789+0000","last_change":"2024-04-25T15:21:07.266826+0000","last_active":"2024-04-25T15:21:07.272789+0000","last_peered":"2024-04-25T15:21:07.272789+0000","last_clean":"2024-04-25T15:21:07.272789+0000","last_became_active":"2024-04-25T15:21:07.266540+0000","last_became_peered":"2024-04-25T15:21:07.266540+0000","last_unstale":"2024-04-25T15:21:07.272789+0000","last_undegraded":"2024-04-25T15:21:07.272789+0000","last_fullsized":"2024-04-25T15:21:07.272789+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258984+0000","last_change":"2024-04-25T15:21:07.258984+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258984+0000","last_undegraded":"2024-04-25T15:21:07.258984+0000","last_fullsized":"2024-04-25T15:21:07.258984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272412+0000","last_change":"2024-04-25T15:21:07.268165+0000","last_active":"2024-04-25T15:21:07.272412+0000","last_peered":"2024-04-25T15:21:07.272412+0000","last_clean":"2024-04-25T15:21:07.272412+0000","last_became_active":"2024-04-25T15:21:07.267821+0000","last_became_peered":"2024-04-25T15:21:07.267821+0000","last_unstale":"2024-04-25T15:21:07.272412+0000","last_undegraded":"2024-04-25T15:21:07.272412+0000","last_fullsized":"2024-04-25T15:21:07.272412+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255378+0000","last_change":"2024-04-25T15:21:07.255378+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255378+0000","last_undegraded":"2024-04-25T15:21:07.255378+0000","last_fullsized":"2024-04-25T15:21:07.255378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.260943+0000","last_change":"2024-04-25T15:21:07.258767+0000","last_active":"2024-04-25T15:21:07.260943+0000","last_peered":"2024-04-25T15:21:07.260943+0000","last_clean":"2024-04-25T15:21:07.260943+0000","last_became_active":"2024-04-25T15:21:07.258461+0000","last_became_peered":"2024-04-25T15:21:07.258461+0000","last_unstale":"2024-04-25T15:21:07.260943+0000","last_undegraded":"2024-04-25T15:21:07.260943+0000","last_fullsized":"2024-04-25T15:21:07.260943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252100+0000","last_change":"2024-04-25T15:21:07.252100+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252100+0000","last_undegraded":"2024-04-25T15:21:07.252100+0000","last_fullsized":"2024-04-25T15:21:07.252100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280453+0000","last_change":"2024-04-25T15:21:07.273310+0000","last_active":"2024-04-25T15:21:07.280453+0000","last_peered":"2024-04-25T15:21:07.280453+0000","last_clean":"2024-04-25T15:21:07.280453+0000","last_became_active":"2024-04-25T15:21:07.272977+0000","last_became_peered":"2024-04-25T15:21:07.272977+0000","last_unstale":"2024-04-25T15:21:07.280453+0000","last_undegraded":"2024-04-25T15:21:07.280453+0000","last_fullsized":"2024-04-25T15:21:07.280453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:29.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253544+0000","last_change":"2024-04-25T15:21:07.253544+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253544+0000","last_undegraded":"2024-04-25T15:21:07.253544+0000","last_fullsized":"2024-04-25T15:21:07.253544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.271664+0000","last_change":"2024-04-25T15:21:07.263740+0000","last_active":"2024-04-25T15:21:07.271664+0000","last_peered":"2024-04-25T15:21:07.271664+0000","last_clean":"2024-04-25T15:21:07.271664+0000","last_became_active":"2024-04-25T15:21:07.262783+0000","last_became_peered":"2024-04-25T15:21:07.262783+0000","last_unstale":"2024-04-25T15:21:07.271664+0000","last_undegraded":"2024-04-25T15:21:07.271664+0000","last_fullsized":"2024-04-25T15:21:07.271664+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258866+0000","last_change":"2024-04-25T15:21:07.258866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258866+0000","last_undegraded":"2024-04-25T15:21:07.258866+0000","last_fullsized":"2024-04-25T15:21:07.258866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272273+0000","last_change":"2024-04-25T15:21:07.267048+0000","last_active":"2024-04-25T15:21:07.272273+0000","last_peered":"2024-04-25T15:21:07.272273+0000","last_clean":"2024-04-25T15:21:07.272273+0000","last_became_active":"2024-04-25T15:21:07.265790+0000","last_became_peered":"2024-04-25T15:21:07.265790+0000","last_unstale":"2024-04-25T15:21:07.272273+0000","last_undegraded":"2024-04-25T15:21:07.272273+0000","last_fullsized":"2024-04-25T15:21:07.272273+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257928+0000","last_change":"2024-04-25T15:21:07.257928+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257928+0000","last_undegraded":"2024-04-25T15:21:07.257928+0000","last_fullsized":"2024-04-25T15:21:07.257928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279451+0000","last_change":"2024-04-25T15:21:07.273306+0000","last_active":"2024-04-25T15:21:07.279451+0000","last_peered":"2024-04-25T15:21:07.279451+0000","last_clean":"2024-04-25T15:21:07.279451+0000","last_became_active":"2024-04-25T15:21:07.272979+0000","last_became_peered":"2024-04-25T15:21:07.272979+0000","last_unstale":"2024-04-25T15:21:07.279451+0000","last_undegraded":"2024-04-25T15:21:07.279451+0000","last_fullsized":"2024-04-25T15:21:07.279451+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:09:56.214534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.262262+0000","last_change":"2024-04-25T15:21:07.262262+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.262262+0000","last_undegraded":"2024-04-25T15:21:07.262262+0000","last_fullsized":"2024-04-25T15:21:07.262262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270661+0000","last_change":"2024-04-25T15:21:07.268388+0000","last_active":"2024-04-25T15:21:07.270661+0000","last_peered":"2024-04-25T15:21:07.270661+0000","last_clean":"2024-04-25T15:21:07.270661+0000","last_became_active":"2024-04-25T15:21:07.267977+0000","last_became_peered":"2024-04-25T15:21:07.267977+0000","last_unstale":"2024-04-25T15:21:07.270661+0000","last_undegraded":"2024-04-25T15:21:07.270661+0000","last_fullsized":"2024-04-25T15:21:07.270661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254905+0000","last_change":"2024-04-25T15:21:07.254905+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254905+0000","last_undegraded":"2024-04-25T15:21:07.254905+0000","last_fullsized":"2024-04-25T15:21:07.254905+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270271+0000","last_change":"2024-04-25T15:21:07.263141+0000","last_active":"2024-04-25T15:21:07.270271+0000","last_peered":"2024-04-25T15:21:07.270271+0000","last_clean":"2024-04-25T15:21:07.270271+0000","last_became_active":"2024-04-25T15:21:07.262765+0000","last_became_peered":"2024-04-25T15:21:07.262765+0000","last_unstale":"2024-04-25T15:21:07.270271+0000","last_undegraded":"2024-04-25T15:21:07.270271+0000","last_fullsized":"2024-04-25T15:21:07.270271+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.252968+0000","last_change":"2024-04-25T15:21:07.252968+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.252968+0000","last_undegraded":"2024-04-25T15:21:07.252968+0000","last_fullsized":"2024-04-25T15:21:07.252968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272311+0000","last_change":"2024-04-25T15:21:07.268503+0000","last_active":"2024-04-25T15:21:07.272311+0000","last_peered":"2024-04-25T15:21:07.272311+0000","last_clean":"2024-04-25T15:21:07.272311+0000","last_became_active":"2024-04-25T15:21:07.268088+0000","last_became_peered":"2024-04-25T15:21:07.268088+0000","last_unstale":"2024-04-25T15:21:07.272311+0000","last_undegraded":"2024-04-25T15:21:07.272311+0000","last_fullsized":"2024-04-25T15:21:07.272311+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:36.404924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254326+0000","last_change":"2024-04-25T15:21:07.254326+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254326+0000","last_undegraded":"2024-04-25T15:21:07.254326+0000","last_fullsized":"2024-04-25T15:21:07.254326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272165+0000","last_change":"2024-04-25T15:21:07.267053+0000","last_active":"2024-04-25T15:21:07.272165+0000","last_peered":"2024-04-25T15:21:07.272165+0000","last_clean":"2024-04-25T15:21:07.272165+0000","last_became_active":"2024-04-25T15:21:07.266690+0000","last_became_peered":"2024-04-25T15:21:07.266690+0000","last_unstale":"2024-04-25T15:21:07.272165+0000","last_undegraded":"2024-04-25T15:21:07.272165+0000","last_fullsized":"2024-04-25T15:21:07.272165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:47.899553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257171+0000","last_change":"2024-04-25T15:21:07.257171+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257171+0000","last_undegraded":"2024-04-25T15:21:07.257171+0000","last_fullsized":"2024-04-25T15:21:07.257171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272200+0000","last_change":"2024-04-25T15:21:07.262699+0000","last_active":"2024-04-25T15:21:07.272200+0000","last_peered":"2024-04-25T15:21:07.272200+0000","last_clean":"2024-04-25T15:21:07.272200+0000","last_became_active":"2024-04-25T15:21:07.262384+0000","last_became_peered":"2024-04-25T15:21:07.262384+0000","last_unstale":"2024-04-25T15:21:07.272200+0000","last_undegraded":"2024-04-25T15:21:07.272200+0000","last_fullsized":"2024-04-25T15:21:07.272200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256033+0000","last_change":"2024-04-25T15:21:07.256033+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256033+0000","last_undegraded":"2024-04-25T15:21:07.256033+0000","last_fullsized":"2024-04-25T15:21:07.256033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274736+0000","last_change":"2024-04-25T15:21:07.270561+0000","last_active":"2024-04-25T15:21:07.274736+0000","last_peered":"2024-04-25T15:21:07.274736+0000","last_clean":"2024-04-25T15:21:07.274736+0000","last_became_active":"2024-04-25T15:21:07.270106+0000","last_became_peered":"2024-04-25T15:21:07.270106+0000","last_unstale":"2024-04-25T15:21:07.274736+0000","last_undegraded":"2024-04-25T15:21:07.274736+0000","last_fullsized":"2024-04-25T15:21:07.274736+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:14:54.724610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256795+0000","last_change":"2024-04-25T15:21:07.256795+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256795+0000","last_undegraded":"2024-04-25T15:21:07.256795+0000","last_fullsized":"2024-04-25T15:21:07.256795+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.280489+0000","last_change":"2024-04-25T15:21:07.273386+0000","last_active":"2024-04-25T15:21:07.280489+0000","last_peered":"2024-04-25T15:21:07.280489+0000","last_clean":"2024-04-25T15:21:07.280489+0000","last_became_active":"2024-04-25T15:21:07.272939+0000","last_became_peered":"2024-04-25T15:21:07.272939+0000","last_unstale":"2024-04-25T15:21:07.280489+0000","last_undegraded":"2024-04-25T15:21:07.280489+0000","last_fullsized":"2024-04-25T15:21:07.280489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:20.595234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.259420+0000","last_change":"2024-04-25T15:21:07.259420+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.259420+0000","last_undegraded":"2024-04-25T15:21:07.259420+0000","last_fullsized":"2024-04-25T15:21:07.259420+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270390+0000","last_change":"2024-04-25T15:21:07.266432+0000","last_active":"2024-04-25T15:21:07.270390+0000","last_peered":"2024-04-25T15:21:07.270390+0000","last_clean":"2024-04-25T15:21:07.270390+0000","last_became_active":"2024-04-25T15:21:07.266089+0000","last_became_peered":"2024-04-25T15:21:07.266089+0000","last_unstale":"2024-04-25T15:21:07.270390+0000","last_undegraded":"2024-04-25T15:21:07.270390+0000","last_fullsized":"2024-04-25T15:21:07.270390+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254229+0000","last_change":"2024-04-25T15:21:07.254229+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254229+0000","last_undegraded":"2024-04-25T15:21:07.254229+0000","last_fullsized":"2024-04-25T15:21:07.254229+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269440+0000","last_change":"2024-04-25T15:21:07.262181+0000","last_active":"2024-04-25T15:21:07.269440+0000","last_peered":"2024-04-25T15:21:07.269440+0000","last_clean":"2024-04-25T15:21:07.269440+0000","last_became_active":"2024-04-25T15:21:07.261828+0000","last_became_peered":"2024-04-25T15:21:07.261828+0000","last_unstale":"2024-04-25T15:21:07.269440+0000","last_undegraded":"2024-04-25T15:21:07.269440+0000","last_fullsized":"2024-04-25T15:21:07.269440+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256308+0000","last_change":"2024-04-25T15:21:07.256308+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256308+0000","last_undegraded":"2024-04-25T15:21:07.256308+0000","last_fullsized":"2024-04-25T15:21:07.256308+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276901+0000","last_change":"2024-04-25T15:21:07.270193+0000","last_active":"2024-04-25T15:21:07.276901+0000","last_peered":"2024-04-25T15:21:07.276901+0000","last_clean":"2024-04-25T15:21:07.276901+0000","last_became_active":"2024-04-25T15:21:07.269650+0000","last_became_peered":"2024-04-25T15:21:07.269650+0000","last_unstale":"2024-04-25T15:21:07.276901+0000","last_undegraded":"2024-04-25T15:21:07.276901+0000","last_fullsized":"2024-04-25T15:21:07.276901+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:17.140514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258937+0000","last_change":"2024-04-25T15:21:07.258937+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258937+0000","last_undegraded":"2024-04-25T15:21:07.258937+0000","last_fullsized":"2024-04-25T15:21:07.258937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.273489+0000","last_change":"2024-04-25T15:21:07.270403+0000","last_active":"2024-04-25T15:21:07.273489+0000","last_peered":"2024-04-25T15:21:07.273489+0000","last_clean":"2024-04-25T15:21:07.273489+0000","last_became_active":"2024-04-25T15:21:07.270082+0000","last_became_peered":"2024-04-25T15:21:07.270082+0000","last_unstale":"2024-04-25T15:21:07.273489+0000","last_undegraded":"2024-04-25T15:21:07.273489+0000","last_fullsized":"2024-04-25T15:21:07.273489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256563+0000","last_change":"2024-04-25T15:21:07.256563+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256563+0000","last_undegraded":"2024-04-25T15:21:07.256563+0000","last_fullsized":"2024-04-25T15:21:07.256563+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276978+0000","last_change":"2024-04-25T15:21:07.270988+0000","last_active":"2024-04-25T15:21:07.276978+0000","last_peered":"2024-04-25T15:21:07.276978+0000","last_clean":"2024-04-25T15:21:07.276978+0000","last_became_active":"2024-04-25T15:21:07.270686+0000","last_became_peered":"2024-04-25T15:21:07.270686+0000","last_unstale":"2024-04-25T15:21:07.276978+0000","last_undegraded":"2024-04-25T15:21:07.276978+0000","last_fullsized":"2024-04-25T15:21:07.276978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:14.556209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253688+0000","last_change":"2024-04-25T15:21:07.253688+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253688+0000","last_undegraded":"2024-04-25T15:21:07.253688+0000","last_fullsized":"2024-04-25T15:21:07.253688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.276948+0000","last_change":"2024-04-25T15:21:07.270720+0000","last_active":"2024-04-25T15:21:07.276948+0000","last_peered":"2024-04-25T15:21:07.276948+0000","last_clean":"2024-04-25T15:21:07.276948+0000","last_became_active":"2024-04-25T15:21:07.270418+0000","last_became_peered":"2024-04-25T15:21:07.270418+0000","last_unstale":"2024-04-25T15:21:07.276948+0000","last_undegraded":"2024-04-25T15:21:07.276948+0000","last_fullsized":"2024-04-25T15:21:07.276948+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:21:23.664366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254153+0000","last_change":"2024-04-25T15:21:07.254153+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254153+0000","last_undegraded":"2024-04-25T15:21:07.254153+0000","last_fullsized":"2024-04-25T15:21:07.254153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274957+0000","last_change":"2024-04-25T15:21:07.270169+0000","last_active":"2024-04-25T15:21:07.274957+0000","last_peered":"2024-04-25T15:21:07.274957+0000","last_clean":"2024-04-25T15:21:07.274957+0000","last_became_active":"2024-04-25T15:21:07.269753+0000","last_became_peered":"2024-04-25T15:21:07.269753+0000","last_unstale":"2024-04-25T15:21:07.274957+0000","last_undegraded":"2024-04-25T15:21:07.274957+0000","last_fullsized":"2024-04-25T15:21:07.274957+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:37.213221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.257535+0000","last_change":"2024-04-25T15:21:07.257535+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.257535+0000","last_undegraded":"2024-04-25T15:21:07.257535+0000","last_fullsized":"2024-04-25T15:21:07.257535+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272859+0000","last_change":"2024-04-25T15:21:07.269932+0000","last_active":"2024-04-25T15:21:07.272859+0000","last_peered":"2024-04-25T15:21:07.272859+0000","last_clean":"2024-04-25T15:21:07.272859+0000","last_became_active":"2024-04-25T15:21:07.269598+0000","last_became_peered":"2024-04-25T15:21:07.269598+0000","last_unstale":"2024-04-25T15:21:07.272859+0000","last_undegraded":"2024-04-25T15:21:07.272859+0000","last_fullsized":"2024-04-25T15:21:07.272859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:22:39.459934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256124+0000","last_change":"2024-04-25T15:21:07.256124+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256124+0000","last_undegraded":"2024-04-25T15:21:07.256124+0000","last_fullsized":"2024-04-25T15:21:07.256124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270365+0000","last_change":"2024-04-25T15:21:07.263451+0000","last_active":"2024-04-25T15:21:07.270365+0000","last_peered":"2024-04-25T15:21:07.270365+0000","last_clean":"2024-04-25T15:21:07.270365+0000","last_became_active":"2024-04-25T15:21:07.263111+0000","last_became_peered":"2024-04-25T15:21:07.263111+0000","last_unstale":"2024-04-25T15:21:07.270365+0000","last_undegraded":"2024-04-25T15:21:07.270365+0000","last_fullsized":"2024-04-25T15:21:07.270365+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.261818+0000","last_change":"2024-04-25T15:21:07.261818+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.261818+0000","last_undegraded":"2024-04-25T15:21:07.261818+0000","last_fullsized":"2024-04-25T15:21:07.261818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.277011+0000","last_change":"2024-04-25T15:21:07.271421+0000","last_active":"2024-04-25T15:21:07.277011+0000","last_peered":"2024-04-25T15:21:07.277011+0000","last_clean":"2024-04-25T15:21:07.277011+0000","last_became_active":"2024-04-25T15:21:07.271136+0000","last_became_peered":"2024-04-25T15:21:07.271136+0000","last_unstale":"2024-04-25T15:21:07.277011+0000","last_undegraded":"2024-04-25T15:21:07.277011+0000","last_fullsized":"2024-04-25T15:21:07.277011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:34.931594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.258775+0000","last_change":"2024-04-25T15:21:07.258775+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.258775+0000","last_undegraded":"2024-04-25T15:21:07.258775+0000","last_fullsized":"2024-04-25T15:21:07.258775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.269969+0000","last_change":"2024-04-25T15:21:07.267201+0000","last_active":"2024-04-25T15:21:07.269969+0000","last_peered":"2024-04-25T15:21:07.269969+0000","last_clean":"2024-04-25T15:21:07.269969+0000","last_became_active":"2024-04-25T15:21:07.266901+0000","last_became_peered":"2024-04-25T15:21:07.266901+0000","last_unstale":"2024-04-25T15:21:07.269969+0000","last_undegraded":"2024-04-25T15:21:07.269969+0000","last_fullsized":"2024-04-25T15:21:07.269969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.253682+0000","last_change":"2024-04-25T15:21:07.253682+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.253682+0000","last_undegraded":"2024-04-25T15:21:07.253682+0000","last_fullsized":"2024-04-25T15:21:07.253682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.270067+0000","last_change":"2024-04-25T15:21:07.265869+0000","last_active":"2024-04-25T15:21:07.270067+0000","last_peered":"2024-04-25T15:21:07.270067+0000","last_clean":"2024-04-25T15:21:07.270067+0000","last_became_active":"2024-04-25T15:21:07.265566+0000","last_became_peered":"2024-04-25T15:21:07.265566+0000","last_unstale":"2024-04-25T15:21:07.270067+0000","last_undegraded":"2024-04-25T15:21:07.270067+0000","last_fullsized":"2024-04-25T15:21:07.270067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.256908+0000","last_change":"2024-04-25T15:21:07.256908+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.256908+0000","last_undegraded":"2024-04-25T15:21:07.256908+0000","last_fullsized":"2024-04-25T15:21:07.256908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272141+0000","last_change":"2024-04-25T15:21:07.267132+0000","last_active":"2024-04-25T15:21:07.272141+0000","last_peered":"2024-04-25T15:21:07.272141+0000","last_clean":"2024-04-25T15:21:07.272141+0000","last_became_active":"2024-04-25T15:21:07.266783+0000","last_became_peered":"2024-04-25T15:21:07.266783+0000","last_unstale":"2024-04-25T15:21:07.272141+0000","last_undegraded":"2024-04-25T15:21:07.272141+0000","last_fullsized":"2024-04-25T15:21:07.272141+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:10.461067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.255866+0000","last_change":"2024-04-25T15:21:07.255866+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.255866+0000","last_undegraded":"2024-04-25T15:21:07.255866+0000","last_fullsized":"2024-04-25T15:21:07.255866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.272473+0000","last_change":"2024-04-25T15:21:07.268347+0000","last_active":"2024-04-25T15:21:07.272473+0000","last_peered":"2024-04-25T15:21:07.272473+0000","last_clean":"2024-04-25T15:21:07.272473+0000","last_became_active":"2024-04-25T15:21:07.268032+0000","last_became_peered":"2024-04-25T15:21:07.268032+0000","last_unstale":"2024-04-25T15:21:07.272473+0000","last_undegraded":"2024-04-25T15:21:07.272473+0000","last_fullsized":"2024-04-25T15:21:07.272473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.260480+0000","last_change":"2024-04-25T15:21:07.260480+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.260480+0000","last_undegraded":"2024-04-25T15:21:07.260480+0000","last_fullsized":"2024-04-25T15:21:07.260480+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.279438+0000","last_change":"2024-04-25T15:21:07.272594+0000","last_active":"2024-04-25T15:21:07.279438+0000","last_peered":"2024-04-25T15:21:07.279438+0000","last_clean":"2024-04-25T15:21:07.279438+0000","last_became_active":"2024-04-25T15:21:07.272274+0000","last_became_peered":"2024-04-25T15:21:07.272274+0000","last_unstale":"2024-04-25T15:21:07.279438+0000","last_undegraded":"2024-04-25T15:21:07.279438+0000","last_fullsized":"2024-04-25T15:21:07.279438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:09.551367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-25T15:21:07.254935+0000","last_change":"2024-04-25T15:21:07.254935+0000","last_active":"2024-04-25T15:21:07.245862+0000","last_peered":"2024-04-25T15:21:07.245862+0000","last_clean":"2024-04-25T15:21:07.245862+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:21:07.254935+0000","last_undegraded":"2024-04-25T15:21:07.254935+0000","last_fullsized":"2024-04-25T15:21:07.254935+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.274974+0000","last_change":"2024-04-25T15:21:07.271612+0000","last_active":"2024-04-25T15:21:07.274974+0000","last_peered":"2024-04-25T15:21:07.274974+0000","last_clean":"2024-04-25T15:21:07.274974+0000","last_became_active":"2024-04-25T15:21:07.271320+0000","last_became_peered":"2024-04-25T15:21:07.271320+0000","last_unstale":"2024-04-25T15:21:07.274974+0000","last_undegraded":"2024-04-25T15:21:07.274974+0000","last_fullsized":"2024-04-25T15:21:07.274974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:42.520351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":141,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T15:21:07.259258+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:21:07.259258+0000","last_peered":"2024-04-25T15:21:07.259258+0000","last_clean":"2024-04-25T15:21:07.259258+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:21:07.259258+0000","last_undegraded":"2024-04-25T15:21:07.259258+0000","last_fullsized":"2024-04-25T15:21:07.259258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1019904,"data_stored":773544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:21:16.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:21:16.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:21:16.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:21:16.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.857+0000 7f72d83b2640 1 Processor -- start 2024-04-25T15:21:16.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.857+0000 7f72d83b2640 1 -- start start 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d83b2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f72d00cc280 0x7f72d0070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d83b2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d01618f0 0x7f72d0161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d83b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72d015e300 0x7f72d015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d83b2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f72d015bcd0 con 0x7f72d015e300 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d83b2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f72d0162240 con 0x7f72d00cc280 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d83b2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f72d00cd490 con 0x7f72d01618f0 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72d015e300 0x7f72d015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:16.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72d015e300 0x7f72d015e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38168/0 (socket says 172.21.15.146:38168) 2024-04-25T15:21:16.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 -- 172.21.15.146:0/2376812609 learned_addr learned my addr 172.21.15.146:0/2376812609 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:16.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 -- 172.21.15.146:0/2376812609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d01618f0 msgr2=0x7f72d0161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:16.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 --2- 172.21.15.146:0/2376812609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d01618f0 0x7f72d0161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 -- 172.21.15.146:0/2376812609 >> v2:172.21.15.177:3300/0 conn(0x7f72d00cc280 msgr2=0x7f72d0070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:16.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 --2- 172.21.15.146:0/2376812609 >> v2:172.21.15.177:3300/0 conn(0x7f72d00cc280 0x7f72d0070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.859+0000 7f72d694f640 1 -- 172.21.15.146:0/2376812609 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d00711d0 con 0x7f72d015e300 2024-04-25T15:21:16.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.860+0000 7f72d694f640 1 --2- 172.21.15.146:0/2376812609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d015e300 0x7f72d015e710 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f72d0071700 tx=0x7f72c802a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd004680c62c39f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:16.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.860+0000 7f72d514c640 1 -- 172.21.15.146:0/2376812609 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8036070 con 0x7f72d015e300 2024-04-25T15:21:16.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.860+0000 7f72d514c640 1 -- 172.21.15.146:0/2376812609 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c8031600 con 0x7f72d015e300 2024-04-25T15:21:16.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.860+0000 7f72d514c640 1 -- 172.21.15.146:0/2376812609 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80319d0 con 0x7f72d015e300 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 -- 172.21.15.146:0/2376812609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d015e300 msgr2=0x7f72d015e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2376812609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d015e300 0x7f72d015e710 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f72d0071700 tx=0x7f72c802a570 comp rx=0 tx=0).stop 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 -- 172.21.15.146:0/2376812609 shutdown_connections 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2376812609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d015e300 0x7f72d015e710 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2376812609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d01618f0 0x7f72d0161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2376812609 >> v2:172.21.15.177:3300/0 conn(0x7f72d00cc280 0x7f72d0070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 -- 172.21.15.146:0/2376812609 >> 172.21.15.146:0/2376812609 conn(0x7f72d00ce0c0 msgr2=0x7f72d00ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 -- 172.21.15.146:0/2376812609 shutdown_connections 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.861+0000 7f72d83b2640 1 -- 172.21.15.146:0/2376812609 wait complete. 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.863+0000 7f72d83b2640 1 Processor -- start 2024-04-25T15:21:16.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.863+0000 7f72d83b2640 1 -- start start 2024-04-25T15:21:16.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d83b2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d00cc280 0x7f72d01f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d83b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d015e300 0x7f72d01e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d83b2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 0x7f72d01e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d83b2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f72d00cce40 con 0x7f72d01618f0 2024-04-25T15:21:16.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d83b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72d00ccb60 con 0x7f72d015e300 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d83b2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f72d00cd120 con 0x7f72d00cc280 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 0x7f72d01e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 0x7f72d01e8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38174/0 (socket says 172.21.15.146:38174) 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 -- 172.21.15.146:0/2000220560 learned_addr learned my addr 172.21.15.146:0/2000220560 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d614e640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d00cc280 0x7f72d01f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d594d640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d015e300 0x7f72d01e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 -- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d00cc280 msgr2=0x7f72d01f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d00cc280 0x7f72d01f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 -- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d015e300 msgr2=0x7f72d01e85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d015e300 0x7f72d01e85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.864+0000 7f72d694f640 1 -- 172.21.15.146:0/2000220560 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d009cd70 con 0x7f72d01618f0 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.865+0000 7f72d694f640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 0x7f72d01e8b00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f72c8034040 tx=0x7f72c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.865+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8041020 con 0x7f72d01618f0 2024-04-25T15:21:16.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.865+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c80077a0 con 0x7f72d01618f0 2024-04-25T15:21:16.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.865+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d01e9360 con 0x7f72d01618f0 2024-04-25T15:21:16.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.865+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8036040 con 0x7f72d01618f0 2024-04-25T15:21:16.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.865+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d01e9920 con 0x7f72d01618f0 2024-04-25T15:21:16.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.866+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d01e90d0 con 0x7f72d01618f0 2024-04-25T15:21:16.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.866+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f72c8044020 con 0x7f72d01618f0 2024-04-25T15:21:16.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.867+0000 7f72c6ffd640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72b40744e0 0x7f72b40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:16.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.867+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f72c80b60b0 con 0x7f72d01618f0 2024-04-25T15:21:16.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.867+0000 7f72d614e640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72b40744e0 0x7f72b40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:16.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.867+0000 7f72d614e640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72b40744e0 0x7f72b40769f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0098e0 tx=0x7f72cc009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:16.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:16.872+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72c804b030 con 0x7f72d01618f0 2024-04-25T15:21:17.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.044+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72d009cb90 con 0x7f72b40744e0 2024-04-25T15:21:17.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.050+0000 7f72c6ffd640 1 -- 172.21.15.146:0/2000220560 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f72d009cb90 con 0x7f72b40744e0 2024-04-25T15:21:17.053 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:17.055 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:21:17.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72b40744e0 msgr2=0x7f72b40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72b40744e0 0x7f72b40769f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0098e0 tx=0x7f72cc009060 comp rx=0 tx=0).stop 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 msgr2=0x7f72d01e8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 0x7f72d01e8b00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f72c8034040 tx=0x7f72c8004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 shutdown_connections 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f72b40744e0 0x7f72b40769f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d01618f0 0x7f72d01e8b00 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d015e300 0x7f72d01e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 --2- 172.21.15.146:0/2000220560 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d00cc280 0x7f72d01f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 >> 172.21.15.146:0/2000220560 conn(0x7f72d00ce0c0 msgr2=0x7f72d00ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 shutdown_connections 2024-04-25T15:21:17.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.053+0000 7f72d83b2640 1 -- 172.21.15.146:0/2000220560 wait complete. 2024-04-25T15:21:17.065 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-25T15:21:16.006416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220944,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753776,"statfs":{"total":773094113280,"available":772867866624,"internally_reserved":0,"allocated":4857856,"data_stored":2761836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001684"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710538+0000","last_change":"2024-04-25T15:21:08.297393+0000","last_active":"2024-04-25T15:21:09.710538+0000","last_peered":"2024-04-25T15:21:09.710538+0000","last_clean":"2024-04-25T15:21:09.710538+0000","last_became_active":"2024-04-25T15:21:08.296982+0000","last_became_peered":"2024-04-25T15:21:08.296982+0000","last_unstale":"2024-04-25T15:21:09.710538+0000","last_undegraded":"2024-04-25T15:21:09.710538+0000","last_fullsized":"2024-04-25T15:21:09.710538+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:43.637351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710380+0000","last_change":"2024-04-25T15:21:08.292304+0000","last_active":"2024-04-25T15:21:09.710380+0000","last_peered":"2024-04-25T15:21:09.710380+0000","last_clean":"2024-04-25T15:21:09.710380+0000","last_became_active":"2024-04-25T15:21:08.291995+0000","last_became_peered":"2024-04-25T15:21:08.291995+0000","last_unstale":"2024-04-25T15:21:09.710380+0000","last_undegraded":"2024-04-25T15:21:09.710380+0000","last_fullsized":"2024-04-25T15:21:09.710380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:24.372982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279286+0000","last_change":"2024-04-25T15:21:08.286797+0000","last_active":"2024-04-25T15:21:09.279286+0000","last_peered":"2024-04-25T15:21:09.279286+0000","last_clean":"2024-04-25T15:21:09.279286+0000","last_became_active":"2024-04-25T15:21:08.286485+0000","last_became_peered":"2024-04-25T15:21:08.286485+0000","last_unstale":"2024-04-25T15:21:09.279286+0000","last_undegraded":"2024-04-25T15:21:09.279286+0000","last_fullsized":"2024-04-25T15:21:09.279286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:46.233688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068139+0000","last_change":"2024-04-25T15:21:08.306823+0000","last_active":"2024-04-25T15:21:10.068139+0000","last_peered":"2024-04-25T15:21:10.068139+0000","last_clean":"2024-04-25T15:21:10.068139+0000","last_became_active":"2024-04-25T15:21:08.306222+0000","last_became_peered":"2024-04-25T15:21:08.306222+0000","last_unstale":"2024-04-25T15:21:10.068139+0000","last_undegraded":"2024-04-25T15:21:10.068139+0000","last_fullsized":"2024-04-25T15:21:10.068139+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:43.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067878+0000","last_change":"2024-04-25T15:21:08.306618+0000","last_active":"2024-04-25T15:21:10.067878+0000","last_peered":"2024-04-25T15:21:10.067878+0000","last_clean":"2024-04-25T15:21:10.067878+0000","last_became_active":"2024-04-25T15:21:08.305996+0000","last_became_peered":"2024-04-25T15:21:08.305996+0000","last_unstale":"2024-04-25T15:21:10.067878+0000","last_undegraded":"2024-04-25T15:21:10.067878+0000","last_fullsized":"2024-04-25T15:21:10.067878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:38.217875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067597+0000","last_change":"2024-04-25T15:21:08.287778+0000","last_active":"2024-04-25T15:21:10.067597+0000","last_peered":"2024-04-25T15:21:10.067597+0000","last_clean":"2024-04-25T15:21:10.067597+0000","last_became_active":"2024-04-25T15:21:08.287441+0000","last_became_peered":"2024-04-25T15:21:08.287441+0000","last_unstale":"2024-04-25T15:21:10.067597+0000","last_undegraded":"2024-04-25T15:21:10.067597+0000","last_fullsized":"2024-04-25T15:21:10.067597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:12.550223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.472773+0000","last_change":"2024-04-25T15:21:08.291651+0000","last_active":"2024-04-25T15:21:09.472773+0000","last_peered":"2024-04-25T15:21:09.472773+0000","last_clean":"2024-04-25T15:21:09.472773+0000","last_became_active":"2024-04-25T15:21:08.291311+0000","last_became_peered":"2024-04-25T15:21:08.291311+0000","last_unstale":"2024-04-25T15:21:09.472773+0000","last_undegraded":"2024-04-25T15:21:09.472773+0000","last_fullsized":"2024-04-25T15:21:09.472773+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:36.142487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.709867+0000","last_change":"2024-04-25T15:21:08.301920+0000","last_active":"2024-04-25T15:21:09.709867+0000","last_peered":"2024-04-25T15:21:09.709867+0000","last_clean":"2024-04-25T15:21:09.709867+0000","last_became_active":"2024-04-25T15:21:08.301675+0000","last_became_peered":"2024-04-25T15:21:08.301675+0000","last_unstale":"2024-04-25T15:21:09.709867+0000","last_undegraded":"2024-04-25T15:21:09.709867+0000","last_fullsized":"2024-04-25T15:21:09.709867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:02.206629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279429+0000","last_change":"2024-04-25T15:21:08.289113+0000","last_active":"2024-04-25T15:21:09.279429+0000","last_peered":"2024-04-25T15:21:09.279429+0000","last_clean":"2024-04-25T15:21:09.279429+0000","last_became_active":"2024-04-25T15:21:08.288819+0000","last_became_peered":"2024-04-25T15:21:08.288819+0000","last_unstale":"2024-04-25T15:21:09.279429+0000","last_undegraded":"2024-04-25T15:21:09.279429+0000","last_fullsized":"2024-04-25T15:21:09.279429+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:16.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473416+0000","last_change":"2024-04-25T15:21:08.281014+0000","last_active":"2024-04-25T15:21:09.473416+0000","last_peered":"2024-04-25T15:21:09.473416+0000","last_clean":"2024-04-25T15:21:09.473416+0000","last_became_active":"2024-04-25T15:21:08.280684+0000","last_became_peered":"2024-04-25T15:21:08.280684+0000","last_unstale":"2024-04-25T15:21:09.473416+0000","last_undegraded":"2024-04-25T15:21:09.473416+0000","last_fullsized":"2024-04-25T15:21:09.473416+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:01.190921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473278+0000","last_change":"2024-04-25T15:21:08.306371+0000","last_active":"2024-04-25T15:21:09.473278+0000","last_peered":"2024-04-25T15:21:09.473278+0000","last_clean":"2024-04-25T15:21:09.473278+0000","last_became_active":"2024-04-25T15:21:08.306019+0000","last_became_peered":"2024-04-25T15:21:08.306019+0000","last_unstale":"2024-04-25T15:21:09.473278+0000","last_undegraded":"2024-04-25T15:21:09.473278+0000","last_fullsized":"2024-04-25T15:21:09.473278+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:47.852603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279084+0000","last_change":"2024-04-25T15:21:08.285021+0000","last_active":"2024-04-25T15:21:09.279084+0000","last_peered":"2024-04-25T15:21:09.279084+0000","last_clean":"2024-04-25T15:21:09.279084+0000","last_became_active":"2024-04-25T15:21:08.284711+0000","last_became_peered":"2024-04-25T15:21:08.284711+0000","last_unstale":"2024-04-25T15:21:09.279084+0000","last_undegraded":"2024-04-25T15:21:09.279084+0000","last_fullsized":"2024-04-25T15:21:09.279084+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:30.487952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.472857+0000","last_change":"2024-04-25T15:21:08.286937+0000","last_active":"2024-04-25T15:21:09.472857+0000","last_peered":"2024-04-25T15:21:09.472857+0000","last_clean":"2024-04-25T15:21:09.472857+0000","last_became_active":"2024-04-25T15:21:08.286604+0000","last_became_peered":"2024-04-25T15:21:08.286604+0000","last_unstale":"2024-04-25T15:21:09.472857+0000","last_undegraded":"2024-04-25T15:21:09.472857+0000","last_fullsized":"2024-04-25T15:21:09.472857+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:16.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280072+0000","last_change":"2024-04-25T15:21:08.305159+0000","last_active":"2024-04-25T15:21:09.280072+0000","last_peered":"2024-04-25T15:21:09.280072+0000","last_clean":"2024-04-25T15:21:09.280072+0000","last_became_active":"2024-04-25T15:21:08.304192+0000","last_became_peered":"2024-04-25T15:21:08.304192+0000","last_unstale":"2024-04-25T15:21:09.280072+0000","last_undegraded":"2024-04-25T15:21:09.280072+0000","last_fullsized":"2024-04-25T15:21:09.280072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:30.487952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473019+0000","last_change":"2024-04-25T15:21:08.283982+0000","last_active":"2024-04-25T15:21:09.473019+0000","last_peered":"2024-04-25T15:21:09.473019+0000","last_clean":"2024-04-25T15:21:09.473019+0000","last_became_active":"2024-04-25T15:21:08.282191+0000","last_became_peered":"2024-04-25T15:21:08.282191+0000","last_unstale":"2024-04-25T15:21:09.473019+0000","last_undegraded":"2024-04-25T15:21:09.473019+0000","last_fullsized":"2024-04-25T15:21:09.473019+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:16.880589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.308931+0000","last_change":"2024-04-25T15:21:08.284380+0000","last_active":"2024-04-25T15:21:09.308931+0000","last_peered":"2024-04-25T15:21:09.308931+0000","last_clean":"2024-04-25T15:21:09.308931+0000","last_became_active":"2024-04-25T15:21:08.281335+0000","last_became_peered":"2024-04-25T15:21:08.281335+0000","last_unstale":"2024-04-25T15:21:09.308931+0000","last_undegraded":"2024-04-25T15:21:09.308931+0000","last_fullsized":"2024-04-25T15:21:09.308931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:05.612596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309503+0000","last_change":"2024-04-25T15:21:08.287680+0000","last_active":"2024-04-25T15:21:09.309503+0000","last_peered":"2024-04-25T15:21:09.309503+0000","last_clean":"2024-04-25T15:21:09.309503+0000","last_became_active":"2024-04-25T15:21:08.287346+0000","last_became_peered":"2024-04-25T15:21:08.287346+0000","last_unstale":"2024-04-25T15:21:09.309503+0000","last_undegraded":"2024-04-25T15:21:09.309503+0000","last_fullsized":"2024-04-25T15:21:09.309503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:30.588564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280760+0000","last_change":"2024-04-25T15:21:08.304554+0000","last_active":"2024-04-25T15:21:09.280760+0000","last_peered":"2024-04-25T15:21:09.280760+0000","last_clean":"2024-04-25T15:21:09.280760+0000","last_became_active":"2024-04-25T15:21:08.304200+0000","last_became_peered":"2024-04-25T15:21:08.304200+0000","last_unstale":"2024-04-25T15:21:09.280760+0000","last_undegraded":"2024-04-25T15:21:09.280760+0000","last_fullsized":"2024-04-25T15:21:09.280760+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:00.766889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068207+0000","last_change":"2024-04-25T15:21:08.284403+0000","last_active":"2024-04-25T15:21:10.068207+0000","last_peered":"2024-04-25T15:21:10.068207+0000","last_clean":"2024-04-25T15:21:10.068207+0000","last_became_active":"2024-04-25T15:21:08.284091+0000","last_became_peered":"2024-04-25T15:21:08.284091+0000","last_unstale":"2024-04-25T15:21:10.068207+0000","last_undegraded":"2024-04-25T15:21:10.068207+0000","last_fullsized":"2024-04-25T15:21:10.068207+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:30.310267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.281034+0000","last_change":"2024-04-25T15:21:08.304782+0000","last_active":"2024-04-25T15:21:09.281034+0000","last_peered":"2024-04-25T15:21:09.281034+0000","last_clean":"2024-04-25T15:21:09.281034+0000","last_became_active":"2024-04-25T15:21:08.304449+0000","last_became_peered":"2024-04-25T15:21:08.304449+0000","last_unstale":"2024-04-25T15:21:09.281034+0000","last_undegraded":"2024-04-25T15:21:09.281034+0000","last_fullsized":"2024-04-25T15:21:09.281034+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:15.685461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473014+0000","last_change":"2024-04-25T15:21:08.284884+0000","last_active":"2024-04-25T15:21:09.473014+0000","last_peered":"2024-04-25T15:21:09.473014+0000","last_clean":"2024-04-25T15:21:09.473014+0000","last_became_active":"2024-04-25T15:21:08.284459+0000","last_became_peered":"2024-04-25T15:21:08.284459+0000","last_unstale":"2024-04-25T15:21:09.473014+0000","last_undegraded":"2024-04-25T15:21:09.473014+0000","last_fullsized":"2024-04-25T15:21:09.473014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:30.588564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280778+0000","last_change":"2024-04-25T15:21:08.303889+0000","last_active":"2024-04-25T15:21:09.280778+0000","last_peered":"2024-04-25T15:21:09.280778+0000","last_clean":"2024-04-25T15:21:09.280778+0000","last_became_active":"2024-04-25T15:21:08.303190+0000","last_became_peered":"2024-04-25T15:21:08.303190+0000","last_unstale":"2024-04-25T15:21:09.280778+0000","last_undegraded":"2024-04-25T15:21:09.280778+0000","last_fullsized":"2024-04-25T15:21:09.280778+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:46.233688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.308986+0000","last_change":"2024-04-25T15:21:08.286318+0000","last_active":"2024-04-25T15:21:09.308986+0000","last_peered":"2024-04-25T15:21:09.308986+0000","last_clean":"2024-04-25T15:21:09.308986+0000","last_became_active":"2024-04-25T15:21:08.284842+0000","last_became_peered":"2024-04-25T15:21:08.284842+0000","last_unstale":"2024-04-25T15:21:09.308986+0000","last_undegraded":"2024-04-25T15:21:09.308986+0000","last_fullsized":"2024-04-25T15:21:09.308986+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:30.487952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067608+0000","last_change":"2024-04-25T15:21:08.298133+0000","last_active":"2024-04-25T15:21:10.067608+0000","last_peered":"2024-04-25T15:21:10.067608+0000","last_clean":"2024-04-25T15:21:10.067608+0000","last_became_active":"2024-04-25T15:21:08.297824+0000","last_became_peered":"2024-04-25T15:21:08.297824+0000","last_unstale":"2024-04-25T15:21:10.067608+0000","last_undegraded":"2024-04-25T15:21:10.067608+0000","last_fullsized":"2024-04-25T15:21:10.067608+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:14.783813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280866+0000","last_change":"2024-04-25T15:21:08.282474+0000","last_active":"2024-04-25T15:21:09.280866+0000","last_peered":"2024-04-25T15:21:09.280866+0000","last_clean":"2024-04-25T15:21:09.280866+0000","last_became_active":"2024-04-25T15:21:08.282144+0000","last_became_peered":"2024-04-25T15:21:08.282144+0000","last_unstale":"2024-04-25T15:21:09.280866+0000","last_undegraded":"2024-04-25T15:21:09.280866+0000","last_fullsized":"2024-04-25T15:21:09.280866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:24.334939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067850+0000","last_change":"2024-04-25T15:21:08.294042+0000","last_active":"2024-04-25T15:21:10.067850+0000","last_peered":"2024-04-25T15:21:10.067850+0000","last_clean":"2024-04-25T15:21:10.067850+0000","last_became_active":"2024-04-25T15:21:08.293697+0000","last_became_peered":"2024-04-25T15:21:08.293697+0000","last_unstale":"2024-04-25T15:21:10.067850+0000","last_undegraded":"2024-04-25T15:21:10.067850+0000","last_fullsized":"2024-04-25T15:21:10.067850+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:36.142487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.281176+0000","last_change":"2024-04-25T15:21:08.292474+0000","last_active":"2024-04-25T15:21:09.281176+0000","last_peered":"2024-04-25T15:21:09.281176+0000","last_clean":"2024-04-25T15:21:09.281176+0000","last_became_active":"2024-04-25T15:21:08.292159+0000","last_became_peered":"2024-04-25T15:21:08.292159+0000","last_unstale":"2024-04-25T15:21:09.281176+0000","last_undegraded":"2024-04-25T15:21:09.281176+0000","last_fullsized":"2024-04-25T15:21:09.281176+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:26.510251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473225+0000","last_change":"2024-04-25T15:21:08.289871+0000","last_active":"2024-04-25T15:21:09.473225+0000","last_peered":"2024-04-25T15:21:09.473225+0000","last_clean":"2024-04-25T15:21:09.473225+0000","last_became_active":"2024-04-25T15:21:08.289530+0000","last_became_peered":"2024-04-25T15:21:08.289530+0000","last_unstale":"2024-04-25T15:21:09.473225+0000","last_undegraded":"2024-04-25T15:21:09.473225+0000","last_fullsized":"2024-04-25T15:21:09.473225+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:27:38.126535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.281198+0000","last_change":"2024-04-25T15:21:08.284988+0000","last_active":"2024-04-25T15:21:09.281198+0000","last_peered":"2024-04-25T15:21:09.281198+0000","last_clean":"2024-04-25T15:21:09.281198+0000","last_became_active":"2024-04-25T15:21:08.284615+0000","last_became_peered":"2024-04-25T15:21:08.284615+0000","last_unstale":"2024-04-25T15:21:09.281198+0000","last_undegraded":"2024-04-25T15:21:09.281198+0000","last_fullsized":"2024-04-25T15:21:09.281198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:53.132862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280576+0000","last_change":"2024-04-25T15:21:08.289239+0000","last_active":"2024-04-25T15:21:09.280576+0000","last_peered":"2024-04-25T15:21:09.280576+0000","last_clean":"2024-04-25T15:21:09.280576+0000","last_became_active":"2024-04-25T15:21:08.288422+0000","last_became_peered":"2024-04-25T15:21:08.288422+0000","last_unstale":"2024-04-25T15:21:09.280576+0000","last_undegraded":"2024-04-25T15:21:09.280576+0000","last_fullsized":"2024-04-25T15:21:09.280576+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:06.829627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309363+0000","last_change":"2024-04-25T15:21:08.286488+0000","last_active":"2024-04-25T15:21:09.309363+0000","last_peered":"2024-04-25T15:21:09.309363+0000","last_clean":"2024-04-25T15:21:09.309363+0000","last_became_active":"2024-04-25T15:21:08.285189+0000","last_became_peered":"2024-04-25T15:21:08.285189+0000","last_unstale":"2024-04-25T15:21:09.309363+0000","last_undegraded":"2024-04-25T15:21:09.309363+0000","last_fullsized":"2024-04-25T15:21:09.309363+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:16.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067346+0000","last_change":"2024-04-25T15:21:08.290718+0000","last_active":"2024-04-25T15:21:10.067346+0000","last_peered":"2024-04-25T15:21:10.067346+0000","last_clean":"2024-04-25T15:21:10.067346+0000","last_became_active":"2024-04-25T15:21:08.290415+0000","last_became_peered":"2024-04-25T15:21:08.290415+0000","last_unstale":"2024-04-25T15:21:10.067346+0000","last_undegraded":"2024-04-25T15:21:10.067346+0000","last_fullsized":"2024-04-25T15:21:10.067346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:30.487952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309191+0000","last_change":"2024-04-25T15:21:08.282983+0000","last_active":"2024-04-25T15:21:09.309191+0000","last_peered":"2024-04-25T15:21:09.309191+0000","last_clean":"2024-04-25T15:21:09.309191+0000","last_became_active":"2024-04-25T15:21:08.282635+0000","last_became_peered":"2024-04-25T15:21:08.282635+0000","last_unstale":"2024-04-25T15:21:09.309191+0000","last_undegraded":"2024-04-25T15:21:09.309191+0000","last_fullsized":"2024-04-25T15:21:09.309191+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:12.550223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280992+0000","last_change":"2024-04-25T15:21:07.273191+0000","last_active":"2024-04-25T15:21:09.280992+0000","last_peered":"2024-04-25T15:21:09.280992+0000","last_clean":"2024-04-25T15:21:09.280992+0000","last_became_active":"2024-04-25T15:21:07.272854+0000","last_became_peered":"2024-04-25T15:21:07.272854+0000","last_unstale":"2024-04-25T15:21:09.280992+0000","last_undegraded":"2024-04-25T15:21:09.280992+0000","last_fullsized":"2024-04-25T15:21:09.280992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:37.213221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067761+0000","last_change":"2024-04-25T15:21:08.289635+0000","last_active":"2024-04-25T15:21:10.067761+0000","last_peered":"2024-04-25T15:21:10.067761+0000","last_clean":"2024-04-25T15:21:10.067761+0000","last_became_active":"2024-04-25T15:21:08.289338+0000","last_became_peered":"2024-04-25T15:21:08.289338+0000","last_unstale":"2024-04-25T15:21:10.067761+0000","last_undegraded":"2024-04-25T15:21:10.067761+0000","last_fullsized":"2024-04-25T15:21:10.067761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:46.233688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279240+0000","last_change":"2024-04-25T15:21:07.268440+0000","last_active":"2024-04-25T15:21:09.279240+0000","last_peered":"2024-04-25T15:21:09.279240+0000","last_clean":"2024-04-25T15:21:09.279240+0000","last_became_active":"2024-04-25T15:21:07.268125+0000","last_became_peered":"2024-04-25T15:21:07.268125+0000","last_unstale":"2024-04-25T15:21:09.279240+0000","last_undegraded":"2024-04-25T15:21:09.279240+0000","last_fullsized":"2024-04-25T15:21:09.279240+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:05.824973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280798+0000","last_change":"2024-04-25T15:21:08.298834+0000","last_active":"2024-04-25T15:21:09.280798+0000","last_peered":"2024-04-25T15:21:09.280798+0000","last_clean":"2024-04-25T15:21:09.280798+0000","last_became_active":"2024-04-25T15:21:08.298321+0000","last_became_peered":"2024-04-25T15:21:08.298321+0000","last_unstale":"2024-04-25T15:21:09.280798+0000","last_undegraded":"2024-04-25T15:21:09.280798+0000","last_fullsized":"2024-04-25T15:21:09.280798+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:36.142487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068134+0000","last_change":"2024-04-25T15:21:07.266826+0000","last_active":"2024-04-25T15:21:10.068134+0000","last_peered":"2024-04-25T15:21:10.068134+0000","last_clean":"2024-04-25T15:21:10.068134+0000","last_became_active":"2024-04-25T15:21:07.266540+0000","last_became_peered":"2024-04-25T15:21:07.266540+0000","last_unstale":"2024-04-25T15:21:10.068134+0000","last_undegraded":"2024-04-25T15:21:10.068134+0000","last_fullsized":"2024-04-25T15:21:10.068134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:14.680807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710538+0000","last_change":"2024-04-25T15:21:08.297325+0000","last_active":"2024-04-25T15:21:09.710538+0000","last_peered":"2024-04-25T15:21:09.710538+0000","last_clean":"2024-04-25T15:21:09.710538+0000","last_became_active":"2024-04-25T15:21:08.296562+0000","last_became_peered":"2024-04-25T15:21:08.296562+0000","last_unstale":"2024-04-25T15:21:09.710538+0000","last_undegraded":"2024-04-25T15:21:09.710538+0000","last_fullsized":"2024-04-25T15:21:09.710538+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:00.770905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280449+0000","last_change":"2024-04-25T15:21:07.268165+0000","last_active":"2024-04-25T15:21:09.280449+0000","last_peered":"2024-04-25T15:21:09.280449+0000","last_clean":"2024-04-25T15:21:09.280449+0000","last_became_active":"2024-04-25T15:21:07.267821+0000","last_became_peered":"2024-04-25T15:21:07.267821+0000","last_unstale":"2024-04-25T15:21:09.280449+0000","last_undegraded":"2024-04-25T15:21:09.280449+0000","last_fullsized":"2024-04-25T15:21:09.280449+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:23.330285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.278968+0000","last_change":"2024-04-25T15:21:08.281831+0000","last_active":"2024-04-25T15:21:09.278968+0000","last_peered":"2024-04-25T15:21:09.278968+0000","last_clean":"2024-04-25T15:21:09.278968+0000","last_became_active":"2024-04-25T15:21:08.281505+0000","last_became_peered":"2024-04-25T15:21:08.281505+0000","last_unstale":"2024-04-25T15:21:09.278968+0000","last_undegraded":"2024-04-25T15:21:09.278968+0000","last_fullsized":"2024-04-25T15:21:09.278968+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:06.099944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473005+0000","last_change":"2024-04-25T15:21:07.258767+0000","last_active":"2024-04-25T15:21:09.473005+0000","last_peered":"2024-04-25T15:21:09.473005+0000","last_clean":"2024-04-25T15:21:09.473005+0000","last_became_active":"2024-04-25T15:21:07.258461+0000","last_became_peered":"2024-04-25T15:21:07.258461+0000","last_unstale":"2024-04-25T15:21:09.473005+0000","last_undegraded":"2024-04-25T15:21:09.473005+0000","last_fullsized":"2024-04-25T15:21:09.473005+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473020+0000","last_change":"2024-04-25T15:21:08.285581+0000","last_active":"2024-04-25T15:21:09.473020+0000","last_peered":"2024-04-25T15:21:09.473020+0000","last_clean":"2024-04-25T15:21:09.473020+0000","last_became_active":"2024-04-25T15:21:08.285300+0000","last_became_peered":"2024-04-25T15:21:08.285300+0000","last_unstale":"2024-04-25T15:21:09.473020+0000","last_undegraded":"2024-04-25T15:21:09.473020+0000","last_fullsized":"2024-04-25T15:21:09.473020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:20.834758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.308973+0000","last_change":"2024-04-25T15:21:07.273310+0000","last_active":"2024-04-25T15:21:09.308973+0000","last_peered":"2024-04-25T15:21:09.308973+0000","last_clean":"2024-04-25T15:21:09.308973+0000","last_became_active":"2024-04-25T15:21:07.272977+0000","last_became_peered":"2024-04-25T15:21:07.272977+0000","last_unstale":"2024-04-25T15:21:09.308973+0000","last_undegraded":"2024-04-25T15:21:09.308973+0000","last_fullsized":"2024-04-25T15:21:09.308973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:02.680499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.281049+0000","last_change":"2024-04-25T15:21:08.304870+0000","last_active":"2024-04-25T15:21:09.281049+0000","last_peered":"2024-04-25T15:21:09.281049+0000","last_clean":"2024-04-25T15:21:09.281049+0000","last_became_active":"2024-04-25T15:21:08.303563+0000","last_became_peered":"2024-04-25T15:21:08.303563+0000","last_unstale":"2024-04-25T15:21:09.281049+0000","last_undegraded":"2024-04-25T15:21:09.281049+0000","last_fullsized":"2024-04-25T15:21:09.281049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:16.770062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067517+0000","last_change":"2024-04-25T15:21:07.263740+0000","last_active":"2024-04-25T15:21:10.067517+0000","last_peered":"2024-04-25T15:21:10.067517+0000","last_clean":"2024-04-25T15:21:10.067517+0000","last_became_active":"2024-04-25T15:21:07.262783+0000","last_became_peered":"2024-04-25T15:21:07.262783+0000","last_unstale":"2024-04-25T15:21:10.067517+0000","last_undegraded":"2024-04-25T15:21:10.067517+0000","last_fullsized":"2024-04-25T15:21:10.067517+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:23.330285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710029+0000","last_change":"2024-04-25T15:21:08.290521+0000","last_active":"2024-04-25T15:21:09.710029+0000","last_peered":"2024-04-25T15:21:09.710029+0000","last_clean":"2024-04-25T15:21:09.710029+0000","last_became_active":"2024-04-25T15:21:08.290119+0000","last_became_peered":"2024-04-25T15:21:08.290119+0000","last_unstale":"2024-04-25T15:21:09.710029+0000","last_undegraded":"2024-04-25T15:21:09.710029+0000","last_fullsized":"2024-04-25T15:21:09.710029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:53.553193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279983+0000","last_change":"2024-04-25T15:21:07.267048+0000","last_active":"2024-04-25T15:21:09.279983+0000","last_peered":"2024-04-25T15:21:09.279983+0000","last_clean":"2024-04-25T15:21:09.279983+0000","last_became_active":"2024-04-25T15:21:07.265790+0000","last_became_peered":"2024-04-25T15:21:07.265790+0000","last_unstale":"2024-04-25T15:21:09.279983+0000","last_undegraded":"2024-04-25T15:21:09.279983+0000","last_fullsized":"2024-04-25T15:21:09.279983+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:05.824973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068176+0000","last_change":"2024-04-25T15:21:08.306414+0000","last_active":"2024-04-25T15:21:10.068176+0000","last_peered":"2024-04-25T15:21:10.068176+0000","last_clean":"2024-04-25T15:21:10.068176+0000","last_became_active":"2024-04-25T15:21:08.306087+0000","last_became_peered":"2024-04-25T15:21:08.306087+0000","last_unstale":"2024-04-25T15:21:10.068176+0000","last_undegraded":"2024-04-25T15:21:10.068176+0000","last_fullsized":"2024-04-25T15:21:10.068176+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:21.599888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710643+0000","last_change":"2024-04-25T15:21:07.273306+0000","last_active":"2024-04-25T15:21:09.710643+0000","last_peered":"2024-04-25T15:21:09.710643+0000","last_clean":"2024-04-25T15:21:09.710643+0000","last_became_active":"2024-04-25T15:21:07.272979+0000","last_became_peered":"2024-04-25T15:21:07.272979+0000","last_unstale":"2024-04-25T15:21:09.710643+0000","last_undegraded":"2024-04-25T15:21:09.710643+0000","last_fullsized":"2024-04-25T15:21:09.710643+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:11.545569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309321+0000","last_change":"2024-04-25T15:21:08.288583+0000","last_active":"2024-04-25T15:21:09.309321+0000","last_peered":"2024-04-25T15:21:09.309321+0000","last_clean":"2024-04-25T15:21:09.309321+0000","last_became_active":"2024-04-25T15:21:08.288282+0000","last_became_peered":"2024-04-25T15:21:08.288282+0000","last_unstale":"2024-04-25T15:21:09.309321+0000","last_undegraded":"2024-04-25T15:21:09.309321+0000","last_fullsized":"2024-04-25T15:21:09.309321+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:36.142487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473475+0000","last_change":"2024-04-25T15:21:07.268388+0000","last_active":"2024-04-25T15:21:09.473475+0000","last_peered":"2024-04-25T15:21:09.473475+0000","last_clean":"2024-04-25T15:21:09.473475+0000","last_became_active":"2024-04-25T15:21:07.267977+0000","last_became_peered":"2024-04-25T15:21:07.267977+0000","last_unstale":"2024-04-25T15:21:09.473475+0000","last_undegraded":"2024-04-25T15:21:09.473475+0000","last_fullsized":"2024-04-25T15:21:09.473475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:05.824973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279195+0000","last_change":"2024-04-25T15:21:08.280957+0000","last_active":"2024-04-25T15:21:09.279195+0000","last_peered":"2024-04-25T15:21:09.279195+0000","last_clean":"2024-04-25T15:21:09.279195+0000","last_became_active":"2024-04-25T15:21:08.280563+0000","last_became_peered":"2024-04-25T15:21:08.280563+0000","last_unstale":"2024-04-25T15:21:09.279195+0000","last_undegraded":"2024-04-25T15:21:09.279195+0000","last_fullsized":"2024-04-25T15:21:09.279195+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:12.550223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473409+0000","last_change":"2024-04-25T15:21:07.263141+0000","last_active":"2024-04-25T15:21:09.473409+0000","last_peered":"2024-04-25T15:21:09.473409+0000","last_clean":"2024-04-25T15:21:09.473409+0000","last_became_active":"2024-04-25T15:21:07.262765+0000","last_became_peered":"2024-04-25T15:21:07.262765+0000","last_unstale":"2024-04-25T15:21:09.473409+0000","last_undegraded":"2024-04-25T15:21:09.473409+0000","last_fullsized":"2024-04-25T15:21:09.473409+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:23.330285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.281299+0000","last_change":"2024-04-25T15:21:08.304853+0000","last_active":"2024-04-25T15:21:09.281299+0000","last_peered":"2024-04-25T15:21:09.281299+0000","last_clean":"2024-04-25T15:21:09.281299+0000","last_became_active":"2024-04-25T15:21:08.304233+0000","last_became_peered":"2024-04-25T15:21:08.304233+0000","last_unstale":"2024-04-25T15:21:09.281299+0000","last_undegraded":"2024-04-25T15:21:09.281299+0000","last_fullsized":"2024-04-25T15:21:09.281299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:34.717070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710186+0000","last_change":"2024-04-25T15:21:07.268503+0000","last_active":"2024-04-25T15:21:09.710186+0000","last_peered":"2024-04-25T15:21:09.710186+0000","last_clean":"2024-04-25T15:21:09.710186+0000","last_became_active":"2024-04-25T15:21:07.268088+0000","last_became_peered":"2024-04-25T15:21:07.268088+0000","last_unstale":"2024-04-25T15:21:09.710186+0000","last_undegraded":"2024-04-25T15:21:09.710186+0000","last_fullsized":"2024-04-25T15:21:09.710186+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:04.582684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279409+0000","last_change":"2024-04-25T15:21:08.287637+0000","last_active":"2024-04-25T15:21:09.279409+0000","last_peered":"2024-04-25T15:21:09.279409+0000","last_clean":"2024-04-25T15:21:09.279409+0000","last_became_active":"2024-04-25T15:21:08.287338+0000","last_became_peered":"2024-04-25T15:21:08.287338+0000","last_unstale":"2024-04-25T15:21:09.279409+0000","last_undegraded":"2024-04-25T15:21:09.279409+0000","last_fullsized":"2024-04-25T15:21:09.279409+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:36.142487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710250+0000","last_change":"2024-04-25T15:21:07.267053+0000","last_active":"2024-04-25T15:21:09.710250+0000","last_peered":"2024-04-25T15:21:09.710250+0000","last_clean":"2024-04-25T15:21:09.710250+0000","last_became_active":"2024-04-25T15:21:07.266690+0000","last_became_peered":"2024-04-25T15:21:07.266690+0000","last_unstale":"2024-04-25T15:21:09.710250+0000","last_undegraded":"2024-04-25T15:21:09.710250+0000","last_fullsized":"2024-04-25T15:21:09.710250+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:29.483298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710083+0000","last_change":"2024-04-25T15:21:08.289738+0000","last_active":"2024-04-25T15:21:09.710083+0000","last_peered":"2024-04-25T15:21:09.710083+0000","last_clean":"2024-04-25T15:21:09.710083+0000","last_became_active":"2024-04-25T15:21:08.289367+0000","last_became_peered":"2024-04-25T15:21:08.289367+0000","last_unstale":"2024-04-25T15:21:09.710083+0000","last_undegraded":"2024-04-25T15:21:09.710083+0000","last_fullsized":"2024-04-25T15:21:09.710083+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:52.747540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067272+0000","last_change":"2024-04-25T15:21:07.262699+0000","last_active":"2024-04-25T15:21:10.067272+0000","last_peered":"2024-04-25T15:21:10.067272+0000","last_clean":"2024-04-25T15:21:10.067272+0000","last_became_active":"2024-04-25T15:21:07.262384+0000","last_became_peered":"2024-04-25T15:21:07.262384+0000","last_unstale":"2024-04-25T15:21:10.067272+0000","last_undegraded":"2024-04-25T15:21:10.067272+0000","last_fullsized":"2024-04-25T15:21:10.067272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:59.762235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309096+0000","last_change":"2024-04-25T15:21:08.284267+0000","last_active":"2024-04-25T15:21:09.309096+0000","last_peered":"2024-04-25T15:21:09.309096+0000","last_clean":"2024-04-25T15:21:09.309096+0000","last_became_active":"2024-04-25T15:21:08.283964+0000","last_became_peered":"2024-04-25T15:21:08.283964+0000","last_unstale":"2024-04-25T15:21:09.309096+0000","last_undegraded":"2024-04-25T15:21:09.309096+0000","last_fullsized":"2024-04-25T15:21:09.309096+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:46.233688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280218+0000","last_change":"2024-04-25T15:21:07.270561+0000","last_active":"2024-04-25T15:21:09.280218+0000","last_peered":"2024-04-25T15:21:09.280218+0000","last_clean":"2024-04-25T15:21:09.280218+0000","last_became_active":"2024-04-25T15:21:07.270106+0000","last_became_peered":"2024-04-25T15:21:07.270106+0000","last_unstale":"2024-04-25T15:21:09.280218+0000","last_undegraded":"2024-04-25T15:21:09.280218+0000","last_fullsized":"2024-04-25T15:21:09.280218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:13.779159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710221+0000","last_change":"2024-04-25T15:21:08.297754+0000","last_active":"2024-04-25T15:21:09.710221+0000","last_peered":"2024-04-25T15:21:09.710221+0000","last_clean":"2024-04-25T15:21:09.710221+0000","last_became_active":"2024-04-25T15:21:08.297024+0000","last_became_peered":"2024-04-25T15:21:08.297024+0000","last_unstale":"2024-04-25T15:21:09.710221+0000","last_undegraded":"2024-04-25T15:21:09.710221+0000","last_fullsized":"2024-04-25T15:21:09.710221+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:24:52.331285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.308950+0000","last_change":"2024-04-25T15:21:07.273386+0000","last_active":"2024-04-25T15:21:09.308950+0000","last_peered":"2024-04-25T15:21:09.308950+0000","last_clean":"2024-04-25T15:21:09.308950+0000","last_became_active":"2024-04-25T15:21:07.272939+0000","last_became_peered":"2024-04-25T15:21:07.272939+0000","last_unstale":"2024-04-25T15:21:09.308950+0000","last_undegraded":"2024-04-25T15:21:09.308950+0000","last_fullsized":"2024-04-25T15:21:09.308950+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710139+0000","last_change":"2024-04-25T15:21:08.284607+0000","last_active":"2024-04-25T15:21:09.710139+0000","last_peered":"2024-04-25T15:21:09.710139+0000","last_clean":"2024-04-25T15:21:09.710139+0000","last_became_active":"2024-04-25T15:21:08.284298+0000","last_became_peered":"2024-04-25T15:21:08.284298+0000","last_unstale":"2024-04-25T15:21:09.710139+0000","last_undegraded":"2024-04-25T15:21:09.710139+0000","last_fullsized":"2024-04-25T15:21:09.710139+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:30.851934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473482+0000","last_change":"2024-04-25T15:21:07.266432+0000","last_active":"2024-04-25T15:21:09.473482+0000","last_peered":"2024-04-25T15:21:09.473482+0000","last_clean":"2024-04-25T15:21:09.473482+0000","last_became_active":"2024-04-25T15:21:07.266089+0000","last_became_peered":"2024-04-25T15:21:07.266089+0000","last_unstale":"2024-04-25T15:21:09.473482+0000","last_undegraded":"2024-04-25T15:21:09.473482+0000","last_fullsized":"2024-04-25T15:21:09.473482+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:59.762235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473317+0000","last_change":"2024-04-25T15:21:08.283667+0000","last_active":"2024-04-25T15:21:09.473317+0000","last_peered":"2024-04-25T15:21:09.473317+0000","last_clean":"2024-04-25T15:21:09.473317+0000","last_became_active":"2024-04-25T15:21:08.283340+0000","last_became_peered":"2024-04-25T15:21:08.283340+0000","last_unstale":"2024-04-25T15:21:09.473317+0000","last_undegraded":"2024-04-25T15:21:09.473317+0000","last_fullsized":"2024-04-25T15:21:09.473317+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:57.113921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280922+0000","last_change":"2024-04-25T15:21:07.262181+0000","last_active":"2024-04-25T15:21:09.280922+0000","last_peered":"2024-04-25T15:21:09.280922+0000","last_clean":"2024-04-25T15:21:09.280922+0000","last_became_active":"2024-04-25T15:21:07.261828+0000","last_became_peered":"2024-04-25T15:21:07.261828+0000","last_unstale":"2024-04-25T15:21:09.280922+0000","last_undegraded":"2024-04-25T15:21:09.280922+0000","last_fullsized":"2024-04-25T15:21:09.280922+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:14.680807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280607+0000","last_change":"2024-04-25T15:21:08.303929+0000","last_active":"2024-04-25T15:21:09.280607+0000","last_peered":"2024-04-25T15:21:09.280607+0000","last_clean":"2024-04-25T15:21:09.280607+0000","last_became_active":"2024-04-25T15:21:08.303479+0000","last_became_peered":"2024-04-25T15:21:08.303479+0000","last_unstale":"2024-04-25T15:21:09.280607+0000","last_undegraded":"2024-04-25T15:21:09.280607+0000","last_fullsized":"2024-04-25T15:21:09.280607+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:12.550223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068270+0000","last_change":"2024-04-25T15:21:07.270193+0000","last_active":"2024-04-25T15:21:10.068270+0000","last_peered":"2024-04-25T15:21:10.068270+0000","last_clean":"2024-04-25T15:21:10.068270+0000","last_became_active":"2024-04-25T15:21:07.269650+0000","last_became_peered":"2024-04-25T15:21:07.269650+0000","last_unstale":"2024-04-25T15:21:10.068270+0000","last_undegraded":"2024-04-25T15:21:10.068270+0000","last_fullsized":"2024-04-25T15:21:10.068270+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:21.986964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068371+0000","last_change":"2024-04-25T15:21:08.287211+0000","last_active":"2024-04-25T15:21:10.068371+0000","last_peered":"2024-04-25T15:21:10.068371+0000","last_clean":"2024-04-25T15:21:10.068371+0000","last_became_active":"2024-04-25T15:21:08.286912+0000","last_became_peered":"2024-04-25T15:21:08.286912+0000","last_unstale":"2024-04-25T15:21:10.068371+0000","last_undegraded":"2024-04-25T15:21:10.068371+0000","last_fullsized":"2024-04-25T15:21:10.068371+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:57:03.685153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068037+0000","last_change":"2024-04-25T15:21:07.270403+0000","last_active":"2024-04-25T15:21:10.068037+0000","last_peered":"2024-04-25T15:21:10.068037+0000","last_clean":"2024-04-25T15:21:10.068037+0000","last_became_active":"2024-04-25T15:21:07.270082+0000","last_became_peered":"2024-04-25T15:21:07.270082+0000","last_unstale":"2024-04-25T15:21:10.068037+0000","last_undegraded":"2024-04-25T15:21:10.068037+0000","last_fullsized":"2024-04-25T15:21:10.068037+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:05.824973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309167+0000","last_change":"2024-04-25T15:21:08.293955+0000","last_active":"2024-04-25T15:21:09.309167+0000","last_peered":"2024-04-25T15:21:09.309167+0000","last_clean":"2024-04-25T15:21:09.309167+0000","last_became_active":"2024-04-25T15:21:08.293631+0000","last_became_peered":"2024-04-25T15:21:08.293631+0000","last_unstale":"2024-04-25T15:21:09.309167+0000","last_undegraded":"2024-04-25T15:21:09.309167+0000","last_fullsized":"2024-04-25T15:21:09.309167+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:05.587338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067938+0000","last_change":"2024-04-25T15:21:07.270988+0000","last_active":"2024-04-25T15:21:10.067938+0000","last_peered":"2024-04-25T15:21:10.067938+0000","last_clean":"2024-04-25T15:21:10.067938+0000","last_became_active":"2024-04-25T15:21:07.270686+0000","last_became_peered":"2024-04-25T15:21:07.270686+0000","last_unstale":"2024-04-25T15:21:10.067938+0000","last_undegraded":"2024-04-25T15:21:10.067938+0000","last_fullsized":"2024-04-25T15:21:10.067938+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:05.728602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279006+0000","last_change":"2024-04-25T15:21:08.280592+0000","last_active":"2024-04-25T15:21:09.279006+0000","last_peered":"2024-04-25T15:21:09.279006+0000","last_clean":"2024-04-25T15:21:09.279006+0000","last_became_active":"2024-04-25T15:21:08.280232+0000","last_became_peered":"2024-04-25T15:21:08.280232+0000","last_unstale":"2024-04-25T15:21:09.279006+0000","last_undegraded":"2024-04-25T15:21:09.279006+0000","last_fullsized":"2024-04-25T15:21:09.279006+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:05.612596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067802+0000","last_change":"2024-04-25T15:21:07.270720+0000","last_active":"2024-04-25T15:21:10.067802+0000","last_peered":"2024-04-25T15:21:10.067802+0000","last_clean":"2024-04-25T15:21:10.067802+0000","last_became_active":"2024-04-25T15:21:07.270418+0000","last_became_peered":"2024-04-25T15:21:07.270418+0000","last_unstale":"2024-04-25T15:21:10.067802+0000","last_undegraded":"2024-04-25T15:21:10.067802+0000","last_fullsized":"2024-04-25T15:21:10.067802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:36:01.297783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.472888+0000","last_change":"2024-04-25T15:21:08.285578+0000","last_active":"2024-04-25T15:21:09.472888+0000","last_peered":"2024-04-25T15:21:09.472888+0000","last_clean":"2024-04-25T15:21:09.472888+0000","last_became_active":"2024-04-25T15:21:08.285221+0000","last_became_peered":"2024-04-25T15:21:08.285221+0000","last_unstale":"2024-04-25T15:21:09.472888+0000","last_undegraded":"2024-04-25T15:21:09.472888+0000","last_fullsized":"2024-04-25T15:21:09.472888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:34.717070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309175+0000","last_change":"2024-04-25T15:21:07.270169+0000","last_active":"2024-04-25T15:21:09.309175+0000","last_peered":"2024-04-25T15:21:09.309175+0000","last_clean":"2024-04-25T15:21:09.309175+0000","last_became_active":"2024-04-25T15:21:07.269753+0000","last_became_peered":"2024-04-25T15:21:07.269753+0000","last_unstale":"2024-04-25T15:21:09.309175+0000","last_undegraded":"2024-04-25T15:21:09.309175+0000","last_fullsized":"2024-04-25T15:21:09.309175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:57.797827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710170+0000","last_change":"2024-04-25T15:21:08.288973+0000","last_active":"2024-04-25T15:21:09.710170+0000","last_peered":"2024-04-25T15:21:09.710170+0000","last_clean":"2024-04-25T15:21:09.710170+0000","last_became_active":"2024-04-25T15:21:08.288655+0000","last_became_peered":"2024-04-25T15:21:08.288655+0000","last_unstale":"2024-04-25T15:21:09.710170+0000","last_undegraded":"2024-04-25T15:21:09.710170+0000","last_fullsized":"2024-04-25T15:21:09.710170+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:55.026031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067541+0000","last_change":"2024-04-25T15:21:07.269932+0000","last_active":"2024-04-25T15:21:10.067541+0000","last_peered":"2024-04-25T15:21:10.067541+0000","last_clean":"2024-04-25T15:21:10.067541+0000","last_became_active":"2024-04-25T15:21:07.269598+0000","last_became_peered":"2024-04-25T15:21:07.269598+0000","last_unstale":"2024-04-25T15:21:10.067541+0000","last_undegraded":"2024-04-25T15:21:10.067541+0000","last_fullsized":"2024-04-25T15:21:10.067541+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:08:39.509951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067298+0000","last_change":"2024-04-25T15:21:08.288648+0000","last_active":"2024-04-25T15:21:10.067298+0000","last_peered":"2024-04-25T15:21:10.067298+0000","last_clean":"2024-04-25T15:21:10.067298+0000","last_became_active":"2024-04-25T15:21:08.288336+0000","last_became_peered":"2024-04-25T15:21:08.288336+0000","last_unstale":"2024-04-25T15:21:10.067298+0000","last_undegraded":"2024-04-25T15:21:10.067298+0000","last_fullsized":"2024-04-25T15:21:10.067298+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:05.612596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473603+0000","last_change":"2024-04-25T15:21:07.263451+0000","last_active":"2024-04-25T15:21:09.473603+0000","last_peered":"2024-04-25T15:21:09.473603+0000","last_clean":"2024-04-25T15:21:09.473603+0000","last_became_active":"2024-04-25T15:21:07.263111+0000","last_became_peered":"2024-04-25T15:21:07.263111+0000","last_unstale":"2024-04-25T15:21:09.473603+0000","last_undegraded":"2024-04-25T15:21:09.473603+0000","last_fullsized":"2024-04-25T15:21:09.473603+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:53:14.680807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309380+0000","last_change":"2024-04-25T15:21:08.285562+0000","last_active":"2024-04-25T15:21:09.309380+0000","last_peered":"2024-04-25T15:21:09.309380+0000","last_clean":"2024-04-25T15:21:09.309380+0000","last_became_active":"2024-04-25T15:21:08.285086+0000","last_became_peered":"2024-04-25T15:21:08.285086+0000","last_unstale":"2024-04-25T15:21:09.309380+0000","last_undegraded":"2024-04-25T15:21:09.309380+0000","last_fullsized":"2024-04-25T15:21:09.309380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:34.717070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.068385+0000","last_change":"2024-04-25T15:21:07.271421+0000","last_active":"2024-04-25T15:21:10.068385+0000","last_peered":"2024-04-25T15:21:10.068385+0000","last_clean":"2024-04-25T15:21:10.068385+0000","last_became_active":"2024-04-25T15:21:07.271136+0000","last_became_peered":"2024-04-25T15:21:07.271136+0000","last_unstale":"2024-04-25T15:21:10.068385+0000","last_undegraded":"2024-04-25T15:21:10.068385+0000","last_fullsized":"2024-04-25T15:21:10.068385+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:13.178854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.710393+0000","last_change":"2024-04-25T15:21:08.304324+0000","last_active":"2024-04-25T15:21:09.710393+0000","last_peered":"2024-04-25T15:21:09.710393+0000","last_clean":"2024-04-25T15:21:09.710393+0000","last_became_active":"2024-04-25T15:21:08.304094+0000","last_became_peered":"2024-04-25T15:21:08.304094+0000","last_unstale":"2024-04-25T15:21:09.710393+0000","last_undegraded":"2024-04-25T15:21:09.710393+0000","last_fullsized":"2024-04-25T15:21:09.710393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:12.115088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279382+0000","last_change":"2024-04-25T15:21:07.267201+0000","last_active":"2024-04-25T15:21:09.279382+0000","last_peered":"2024-04-25T15:21:09.279382+0000","last_clean":"2024-04-25T15:21:09.279382+0000","last_became_active":"2024-04-25T15:21:07.266901+0000","last_became_peered":"2024-04-25T15:21:07.266901+0000","last_unstale":"2024-04-25T15:21:09.279382+0000","last_undegraded":"2024-04-25T15:21:09.279382+0000","last_fullsized":"2024-04-25T15:21:09.279382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:07.714065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279173+0000","last_change":"2024-04-25T15:21:08.285933+0000","last_active":"2024-04-25T15:21:09.279173+0000","last_peered":"2024-04-25T15:21:09.279173+0000","last_clean":"2024-04-25T15:21:09.279173+0000","last_became_active":"2024-04-25T15:21:08.285624+0000","last_became_peered":"2024-04-25T15:21:08.285624+0000","last_unstale":"2024-04-25T15:21:09.279173+0000","last_undegraded":"2024-04-25T15:21:09.279173+0000","last_fullsized":"2024-04-25T15:21:09.279173+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:05.587338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279153+0000","last_change":"2024-04-25T15:21:07.265869+0000","last_active":"2024-04-25T15:21:09.279153+0000","last_peered":"2024-04-25T15:21:09.279153+0000","last_clean":"2024-04-25T15:21:09.279153+0000","last_became_active":"2024-04-25T15:21:07.265566+0000","last_became_peered":"2024-04-25T15:21:07.265566+0000","last_unstale":"2024-04-25T15:21:09.279153+0000","last_undegraded":"2024-04-25T15:21:09.279153+0000","last_fullsized":"2024-04-25T15:21:09.279153+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:59.762235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:10.067531+0000","last_change":"2024-04-25T15:21:08.285651+0000","last_active":"2024-04-25T15:21:10.067531+0000","last_peered":"2024-04-25T15:21:10.067531+0000","last_clean":"2024-04-25T15:21:10.067531+0000","last_became_active":"2024-04-25T15:21:08.285308+0000","last_became_peered":"2024-04-25T15:21:08.285308+0000","last_unstale":"2024-04-25T15:21:10.067531+0000","last_undegraded":"2024-04-25T15:21:10.067531+0000","last_fullsized":"2024-04-25T15:21:10.067531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:05.587338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280715+0000","last_change":"2024-04-25T15:21:07.267132+0000","last_active":"2024-04-25T15:21:09.280715+0000","last_peered":"2024-04-25T15:21:09.280715+0000","last_clean":"2024-04-25T15:21:09.280715+0000","last_became_active":"2024-04-25T15:21:07.266783+0000","last_became_peered":"2024-04-25T15:21:07.266783+0000","last_unstale":"2024-04-25T15:21:09.280715+0000","last_undegraded":"2024-04-25T15:21:09.280715+0000","last_fullsized":"2024-04-25T15:21:09.280715+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:52.358895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.280503+0000","last_change":"2024-04-25T15:21:08.305265+0000","last_active":"2024-04-25T15:21:09.280503+0000","last_peered":"2024-04-25T15:21:09.280503+0000","last_clean":"2024-04-25T15:21:09.280503+0000","last_became_active":"2024-04-25T15:21:08.304564+0000","last_became_peered":"2024-04-25T15:21:08.304564+0000","last_unstale":"2024-04-25T15:21:09.280503+0000","last_undegraded":"2024-04-25T15:21:09.280503+0000","last_fullsized":"2024-04-25T15:21:09.280503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:05.587338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.279896+0000","last_change":"2024-04-25T15:21:07.268347+0000","last_active":"2024-04-25T15:21:09.279896+0000","last_peered":"2024-04-25T15:21:09.279896+0000","last_clean":"2024-04-25T15:21:09.279896+0000","last_became_active":"2024-04-25T15:21:07.268032+0000","last_became_peered":"2024-04-25T15:21:07.268032+0000","last_unstale":"2024-04-25T15:21:09.279896+0000","last_undegraded":"2024-04-25T15:21:09.279896+0000","last_fullsized":"2024-04-25T15:21:09.279896+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:59.762235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309557+0000","last_change":"2024-04-25T15:21:08.286290+0000","last_active":"2024-04-25T15:21:09.309557+0000","last_peered":"2024-04-25T15:21:09.309557+0000","last_clean":"2024-04-25T15:21:09.309557+0000","last_became_active":"2024-04-25T15:21:08.285958+0000","last_became_peered":"2024-04-25T15:21:08.285958+0000","last_unstale":"2024-04-25T15:21:09.309557+0000","last_undegraded":"2024-04-25T15:21:09.309557+0000","last_fullsized":"2024-04-25T15:21:09.309557+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:16.880589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.709895+0000","last_change":"2024-04-25T15:21:07.272594+0000","last_active":"2024-04-25T15:21:09.709895+0000","last_peered":"2024-04-25T15:21:09.709895+0000","last_clean":"2024-04-25T15:21:09.709895+0000","last_became_active":"2024-04-25T15:21:07.272274+0000","last_became_peered":"2024-04-25T15:21:07.272274+0000","last_unstale":"2024-04-25T15:21:09.709895+0000","last_undegraded":"2024-04-25T15:21:09.709895+0000","last_fullsized":"2024-04-25T15:21:09.709895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:04.607942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.473434+0000","last_change":"2024-04-25T15:21:08.291062+0000","last_active":"2024-04-25T15:21:09.473434+0000","last_peered":"2024-04-25T15:21:09.473434+0000","last_clean":"2024-04-25T15:21:09.473434+0000","last_became_active":"2024-04-25T15:21:08.290067+0000","last_became_peered":"2024-04-25T15:21:08.290067+0000","last_unstale":"2024-04-25T15:21:09.473434+0000","last_undegraded":"2024-04-25T15:21:09.473434+0000","last_fullsized":"2024-04-25T15:21:09.473434+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:07.245862+0000","last_clean_scrub_stamp":"2024-04-25T15:21:07.245862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:02.206629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309213+0000","last_change":"2024-04-25T15:21:07.271612+0000","last_active":"2024-04-25T15:21:09.309213+0000","last_peered":"2024-04-25T15:21:09.309213+0000","last_clean":"2024-04-25T15:21:09.309213+0000","last_became_active":"2024-04-25T15:21:07.271320+0000","last_became_peered":"2024-04-25T15:21:07.271320+0000","last_unstale":"2024-04-25T15:21:09.309213+0000","last_undegraded":"2024-04-25T15:21:09.309213+0000","last_fullsized":"2024-04-25T15:21:09.309213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:21:06.241208+0000","last_clean_scrub_stamp":"2024-04-25T15:21:06.241208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:34.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":145,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-25T15:21:09.309329+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:21:09.309329+0000","last_peered":"2024-04-25T15:21:09.309329+0000","last_clean":"2024-04-25T15:21:09.309329+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:21:09.309329+0000","last_undegraded":"2024-04-25T15:21:09.309329+0000","last_fullsized":"2024-04-25T15:21:09.309329+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1060864,"data_stored":789085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":197147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":197639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":198681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":787437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:21:17.069 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:21:17.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:21:17.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.219+0000 7fce8151f640 1 Processor -- start 2024-04-25T15:21:17.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.220+0000 7fce8151f640 1 -- start start 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce8151f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce7c0af5d0 0x7fce7c0af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce8151f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7c0cefa0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce8151f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7c0b1f00 con 0x7fce7c09b2b0 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce8151f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fce7c0b2830 con 0x7fce7c0af5d0 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce7affd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce7c0af5d0 0x7fce7c0af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce7b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fce7c0b2cc0 0x7fce7c0b30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54298/0 (socket says 172.21.15.146:54298) 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce7b7fe640 1 -- 172.21.15.146:0/4110687214 learned_addr learned my addr 172.21.15.146:0/4110687214 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce7affd640 1 -- 172.21.15.146:0/4110687214 >> v1:172.21.15.177:6789/0 conn(0x7fce7c09b2b0 legacy=0x7fce7c09b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce7affd640 1 -- 172.21.15.146:0/4110687214 >> v1:172.21.15.146:6789/0 conn(0x7fce7c0b2cc0 legacy=0x7fce7c0b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce79ffb640 1 -- 172.21.15.146:0/4110687214 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045699383 0 0) 0x7fce7c0cefa0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.221+0000 7fce7affd640 1 -- 172.21.15.146:0/4110687214 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c0b21e0 con 0x7fce7c0af5d0 2024-04-25T15:21:17.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.222+0000 7fce7affd640 1 --2- 172.21.15.146:0/4110687214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0af5d0 0x7fce7c0af9e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0b29b0 tx=0x7fce6c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7cb95b9ee7ad332 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:17.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.222+0000 7fce79ffb640 1 -- 172.21.15.146:0/4110687214 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c007740 con 0x7fce7c0af5d0 2024-04-25T15:21:17.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.222+0000 7fce79ffb640 1 -- 172.21.15.146:0/4110687214 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce6c0078e0 con 0x7fce7c0af5d0 2024-04-25T15:21:17.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.222+0000 7fce79ffb640 1 -- 172.21.15.146:0/4110687214 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c007cb0 con 0x7fce7c0af5d0 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 -- 172.21.15.146:0/4110687214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0af5d0 msgr2=0x7fce7c0af9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 --2- 172.21.15.146:0/4110687214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0af5d0 0x7fce7c0af9e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0b29b0 tx=0x7fce6c02a6f0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 -- 172.21.15.146:0/4110687214 shutdown_connections 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 --2- 172.21.15.146:0/4110687214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0af5d0 0x7fce7c0af9e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 -- 172.21.15.146:0/4110687214 >> 172.21.15.146:0/4110687214 conn(0x7fce7c0b7490 msgr2=0x7fce7c0b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 -- 172.21.15.146:0/4110687214 shutdown_connections 2024-04-25T15:21:17.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.223+0000 7fce8151f640 1 -- 172.21.15.146:0/4110687214 wait complete. 2024-04-25T15:21:17.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.225+0000 7fce8151f640 1 Processor -- start 2024-04-25T15:21:17.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.225+0000 7fce8151f640 1 -- start start 2024-04-25T15:21:17.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce8151f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7c09b2b0 0x7fce7c071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce8151f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce7c0af5d0 0x7fce7c0cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce8151f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 0x7fce7c0caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce8151f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c0ce1d0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce8151f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c0b1960 con 0x7fce7c09b2b0 2024-04-25T15:21:17.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce8151f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fce7c0ce350 con 0x7fce7c0af5d0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce7b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 0x7fce7c0caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce7b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 0x7fce7c0caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38196/0 (socket says 172.21.15.146:38196) 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce7b7fe640 1 -- 172.21.15.146:0/3720796582 learned_addr learned my addr 172.21.15.146:0/3720796582 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce7affd640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7c09b2b0 0x7fce7c071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.226+0000 7fce7a7fc640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce7c0af5d0 0x7fce7c0cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce7b7fe640 1 -- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce7c0af5d0 msgr2=0x7fce7c0cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce7b7fe640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce7c0af5d0 0x7fce7c0cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce7b7fe640 1 -- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7c09b2b0 msgr2=0x7fce7c071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce7b7fe640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7c09b2b0 0x7fce7c071900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce7b7fe640 1 -- 172.21.15.146:0/3720796582 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c0c99b0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce7b7fe640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 0x7fce7c0caa30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fce7400c440 tx=0x7fce7400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce740074e0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce740047e0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7c0c9c90 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74004bd0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.227+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7c0ca250 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.228+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce7c0507b0 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.229+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fce74011070 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.229+0000 7fce637fe640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fce680744b0 0x7fce680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.229+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fce74094e00 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.229+0000 7fce7affd640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fce680744b0 0x7fce680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.230+0000 7fce7affd640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fce680744b0 0x7fce680769c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fce6c034000 tx=0x7fce6c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:17.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.234+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce7405ee70 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.481+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fce7c0b0f50 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.483+0000 7fce637fe640 1 -- 172.21.15.146:0/3720796582 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fce74063170 con 0x7fce7c0b2cc0 2024-04-25T15:21:17.484 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-04-25T15:21:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmlczBJhAApZj6jMADYxBHeBvhaprLDg== 2024-04-25T15:21:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: caps mds = "allow" 2024-04-25T15:21:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: caps mon = "allow r" 2024-04-25T15:21:17.484 INFO:teuthology.orchestra.run.smithi146.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:21:17.485 INFO:teuthology.orchestra.run.smithi146.stderr:updated caps for client.0 2024-04-25T15:21:17.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fce680744b0 msgr2=0x7fce680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fce680744b0 0x7fce680769c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fce6c034000 tx=0x7fce6c02f000 comp rx=0 tx=0).stop 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 msgr2=0x7fce7c0caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 0x7fce7c0caa30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fce7400c440 tx=0x7fce7400c900 comp rx=0 tx=0).stop 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 shutdown_connections 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fce680744b0 0x7fce680769c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce7c0b2cc0 0x7fce7c0caa30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce7c0af5d0 0x7fce7c0cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 --2- 172.21.15.146:0/3720796582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7c09b2b0 0x7fce7c071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.485+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 >> 172.21.15.146:0/3720796582 conn(0x7fce7c0b7490 msgr2=0x7fce7c09b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.486+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 shutdown_connections 2024-04-25T15:21:17.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.486+0000 7fce8151f640 1 -- 172.21.15.146:0/3720796582 wait complete. 2024-04-25T15:21:17.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:21:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.604+0000 7f34ad1f8640 1 Processor -- start 2024-04-25T15:21:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.604+0000 7f34ad1f8640 1 -- start start 2024-04-25T15:21:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34ad1f8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f34a80cc280 0x7f34a8070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34ad1f8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34a806c8c0 0x7f34a806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34ad1f8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a80cd490 con 0x7f34a81618f0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34ad1f8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f34a815bcd0 con 0x7f34a806c8c0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34ad1f8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f34a80711d0 con 0x7f34a80cc280 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34a806c8c0 0x7f34a806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34a806c8c0 0x7f34a806ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38206/0 (socket says 172.21.15.146:38206) 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 -- 172.21.15.146:0/2031522122 learned_addr learned my addr 172.21.15.146:0/2031522122 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a6d9d640 1 --2- 172.21.15.146:0/2031522122 >> v2:172.21.15.177:3300/0 conn(0x7f34a80cc280 0x7f34a8070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 -- 172.21.15.146:0/2031522122 >> v2:172.21.15.177:3300/0 conn(0x7f34a80cc280 msgr2=0x7f34a8070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 --2- 172.21.15.146:0/2031522122 >> v2:172.21.15.177:3300/0 conn(0x7f34a80cc280 0x7f34a8070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 -- 172.21.15.146:0/2031522122 >> v1:172.21.15.177:6789/0 conn(0x7f34a81618f0 legacy=0x7f34a8161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.606+0000 7f34a759e640 1 -- 172.21.15.146:0/2031522122 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a80cc880 con 0x7f34a806c8c0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.607+0000 7f34a759e640 1 --2- 172.21.15.146:0/2031522122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a806c8c0 0x7f34a806ccd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f34a8071700 tx=0x7f34a002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf892831ad3a582d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.607+0000 7f34a5d9b640 1 -- 172.21.15.146:0/2031522122 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0036070 con 0x7f34a806c8c0 2024-04-25T15:21:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.607+0000 7f34a5d9b640 1 -- 172.21.15.146:0/2031522122 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a0031600 con 0x7f34a806c8c0 2024-04-25T15:21:17.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.607+0000 7f34a5d9b640 1 -- 172.21.15.146:0/2031522122 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a00319d0 con 0x7f34a806c8c0 2024-04-25T15:21:17.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.607+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2031522122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a806c8c0 msgr2=0x7f34a806ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.607+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2031522122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a806c8c0 0x7f34a806ccd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f34a8071700 tx=0x7f34a002a610 comp rx=0 tx=0).stop 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.608+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2031522122 shutdown_connections 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.608+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2031522122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a806c8c0 0x7f34a806ccd0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.608+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2031522122 >> v2:172.21.15.177:3300/0 conn(0x7f34a80cc280 0x7f34a8070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.608+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2031522122 >> 172.21.15.146:0/2031522122 conn(0x7f34a80ce0c0 msgr2=0x7f34a80ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.608+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2031522122 shutdown_connections 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.608+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2031522122 wait complete. 2024-04-25T15:21:17.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.610+0000 7f34ad1f8640 1 Processor -- start 2024-04-25T15:21:17.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.610+0000 7f34ad1f8640 1 -- start start 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34ad1f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a806c8c0 0x7f34a80b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34ad1f8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 0x7f34a80b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34ad1f8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34a81618f0 0x7f34a81f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34ad1f8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f34a80cd120 con 0x7f34a80cc280 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34ad1f8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34a80cce40 con 0x7f34a806c8c0 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34ad1f8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f34a80b0940 con 0x7f34a81618f0 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 0x7f34a80b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 0x7f34a80b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38220/0 (socket says 172.21.15.146:38220) 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 -- 172.21.15.146:0/2209625300 learned_addr learned my addr 172.21.15.146:0/2209625300 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a6d9d640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a806c8c0 0x7f34a80b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a759e640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34a81618f0 0x7f34a81f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 -- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34a81618f0 msgr2=0x7f34a81f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34a81618f0 0x7f34a81f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 -- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a806c8c0 msgr2=0x7f34a80b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a806c8c0 0x7f34a80b4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a659c640 1 -- 172.21.15.146:0/2209625300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a80b56f0 con 0x7f34a80cc280 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a6d9d640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a806c8c0 0x7f34a80b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.611+0000 7f34a759e640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34a81618f0 0x7f34a81f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.612+0000 7f34a659c640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 0x7f34a80b2740 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c3f0 tx=0x7f349c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.612+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c0052d0 con 0x7f34a80cc280 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.612+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f349c005490 con 0x7f34a80cc280 2024-04-25T15:21:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.612+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a80b59d0 con 0x7f34a80cc280 2024-04-25T15:21:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.612+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c007260 con 0x7f34a80cc280 2024-04-25T15:21:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.612+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a80b5f90 con 0x7f34a80cc280 2024-04-25T15:21:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.613+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a806a0e0 con 0x7f34a80cc280 2024-04-25T15:21:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.613+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f349c007400 con 0x7f34a80cc280 2024-04-25T15:21:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.614+0000 7f348f7fe640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f34940744b0 0x7f34940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.614+0000 7f34a6d9d640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f34940744b0 0x7f34940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.614+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f349c094900 con 0x7f34a80cc280 2024-04-25T15:21:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.614+0000 7f34a6d9d640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f34940744b0 0x7f34940769c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3498009af0 tx=0x7f3498008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.618+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f349c05ee80 con 0x7f34a80cc280 2024-04-25T15:21:17.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.828+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34a806fbb0 con 0x7f34a80cc280 2024-04-25T15:21:17.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.828+0000 7f348f7fe640 1 -- 172.21.15.146:0/2209625300 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5187 (secure 0 0 0) 0x7f349c05e780 con 0x7f34a80cc280 2024-04-25T15:21:17.829 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:17.829 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:10.651740+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5556":{"gid":5556,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6835/539736758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":539736758},{"type":"v1","addr":"172.21.15.177:6835","nonce":539736758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5556,"qdb_cluster":[5556]},"id":2}]} 2024-04-25T15:21:17.829 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 17 2024-04-25T15:21:17.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f34940744b0 msgr2=0x7f34940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f34940744b0 0x7f34940769c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3498009af0 tx=0x7f3498008040 comp rx=0 tx=0).stop 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 msgr2=0x7f34a80b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 0x7f34a80b2740 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c3f0 tx=0x7f349c00c8b0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 shutdown_connections 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f34940744b0 0x7f34940769c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34a81618f0 0x7f34a81f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a80cc280 0x7f34a80b2740 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 --2- 172.21.15.146:0/2209625300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a806c8c0 0x7f34a80b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 >> 172.21.15.146:0/2209625300 conn(0x7f34a80ce0c0 msgr2=0x7f34a80af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 shutdown_connections 2024-04-25T15:21:17.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:17.831+0000 7f34ad1f8640 1 -- 172.21.15.146:0/2209625300 wait complete. 2024-04-25T15:21:18.246 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:21:07.618988+0000', 'modified': '2024-04-25T15:21:10.651740+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5556}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5556': {'gid': 5556, 'name': 'b', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.177:6835/539736758', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6833', 'nonce': 539736758}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 539736758}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5556, 'qdb_cluster': [5556]} 2024-04-25T15:21:18.246 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:21:18.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T15:21:18.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.357+0000 7fc1b8f6b640 1 Processor -- start 2024-04-25T15:21:18.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.357+0000 7fc1b8f6b640 1 -- start start 2024-04-25T15:21:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.358+0000 7fc1b8f6b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1b40b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.358+0000 7fc1b8f6b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1b40b2bb0 con 0x7fc1b406da30 2024-04-25T15:21:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.358+0000 7fc1b8f6b640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1b406def0 con 0x7fc1b406cf60 2024-04-25T15:21:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.358+0000 7fc1b8f6b640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b40b1790 con 0x7fc1b415e060 2024-04-25T15:21:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.358+0000 7fc1b259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc1b406da30 0x7fc1b40cc060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54312/0 (socket says 172.21.15.146:54312) 2024-04-25T15:21:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.358+0000 7fc1b259c640 1 -- 172.21.15.146:0/2195731606 learned_addr learned my addr 172.21.15.146:0/2195731606 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1b40b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1915523575 0 0) 0x7fc1b40b2bb0 con 0x7fc1b406da30 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1a0003520 con 0x7fc1b406da30 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1622354897 0 0) 0x7fc1a0003520 con 0x7fc1b406da30 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 msgr2=0x7fc1b40b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1b40b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 >> v1:172.21.15.177:6789/0 conn(0x7fc1b406cf60 legacy=0x7fc1b40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b40b11d0 con 0x7fc1b406da30 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fc1a4001670 con 0x7fc1b406da30 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 >> v1:172.21.15.146:6789/0 conn(0x7fc1b406da30 legacy=0x7fc1b40cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1a00039e0 0x7fc1a0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1a0004800 0x7fc1a0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1a0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b40b1790 con 0x7fc1a0004800 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a000baf0 con 0x7fc1a00039e0 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a000bcf0 con 0x7fc1b415e060 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1a0004800 0x7fc1a0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b259c640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1a00039e0 0x7fc1a0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1a0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 msgr2=0x7fc1a0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1a0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1a00039e0 msgr2=0x7fc1a0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1a00039e0 0x7fc1a0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/2195731606 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a0003520 con 0x7fc1a0004800 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.359+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b415e060 0x7fc1a0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.360+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1a0004800 0x7fc1a0004c10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009a90 tx=0x7fc1a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618268d36b57f241 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.360+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c036070 con 0x7fc1a0004800 2024-04-25T15:21:18.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.360+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c031400 con 0x7fc1a0004800 2024-04-25T15:21:18.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.360+0000 7fc1b1d9b640 1 -- 172.21.15.146:0/2195731606 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c009690 con 0x7fc1a0004800 2024-04-25T15:21:18.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2195731606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1a0004800 msgr2=0x7fc1a0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2195731606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1a0004800 0x7fc1a0004c10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009a90 tx=0x7fc1a0005830 comp rx=0 tx=0).stop 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/2195731606 reap_dead start 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2195731606 shutdown_connections 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2195731606 >> 172.21.15.146:0/2195731606 conn(0x7fc1b409b2b0 msgr2=0x7fc1b40b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2195731606 shutdown_connections 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.361+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2195731606 wait complete. 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.363+0000 7fc1b8f6b640 1 Processor -- start 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.363+0000 7fc1b8f6b640 1 -- start start 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.364+0000 7fc1b8f6b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b4155690 0x7fc1b4156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.364+0000 7fc1b8f6b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1b4153e80 0x7fc1b40cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.364+0000 7fc1b8f6b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 0x7fc1b40cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.364+0000 7fc1b8f6b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b40b3370 con 0x7fc1b40cfb20 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.364+0000 7fc1b8f6b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b40b2e50 con 0x7fc1b4153e80 2024-04-25T15:21:18.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.364+0000 7fc1b8f6b640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b40d26a0 con 0x7fc1b4155690 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 0x7fc1b40cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 0x7fc1b40cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38240/0 (socket says 172.21.15.146:38240) 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/2253557601 learned_addr learned my addr 172.21.15.146:0/2253557601 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b259c640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b4155690 0x7fc1b4156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b4155690 msgr2=0x7fc1b4156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b4155690 0x7fc1b4156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1b4153e80 0x7fc1b40cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1b4153e80 msgr2=0x7fc1b40cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1b4153e80 0x7fc1b40cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/2253557601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b41545f0 con 0x7fc1b40cfb20 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 0x7fc1b40cff30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40726b0 tx=0x7fc1ac00ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1ab7fe640 1 -- 172.21.15.146:0/2253557601 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac003b80 con 0x7fc1b40cfb20 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1ab7fe640 1 -- 172.21.15.146:0/2253557601 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1ac003d40 con 0x7fc1b40cfb20 2024-04-25T15:21:18.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1ab7fe640 1 -- 172.21.15.146:0/2253557601 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac006b50 con 0x7fc1b40cfb20 2024-04-25T15:21:18.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b41548d0 con 0x7fc1b40cfb20 2024-04-25T15:21:18.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.365+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b4154e90 con 0x7fc1b40cfb20 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.367+0000 7fc1ab7fe640 1 -- 172.21.15.146:0/2253557601 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc1ac01a020 con 0x7fc1b40cfb20 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.367+0000 7fc1ab7fe640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc1940744b0 0x7fc1940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.367+0000 7fc1ab7fe640 1 -- 172.21.15.146:0/2253557601 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc1ac0948e0 con 0x7fc1b40cfb20 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.367+0000 7fc1b259c640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc1940744b0 0x7fc1940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.368+0000 7fc1b259c640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc1940744b0 0x7fc1940769c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4000c00 tx=0x7fc1a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.368+0000 7fc1bfe18740 1 Processor -- start 2024-04-25T15:21:18.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.369+0000 7fc1bfe18740 1 -- start start 2024-04-25T15:21:18.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 0x5618901f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe0d590 0x5618901d0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561890134740 con 0x56188fe053b0 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5618900d03a0 con 0x56188fe2e270 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x561890130700 con 0x56188fe0d590 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 0x5618901f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 0x5618901f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38250/0 (socket says 172.21.15.146:38250) 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/3292229557 learned_addr learned my addr 172.21.15.146:0/3292229557 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1b259c640 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe0d590 0x5618901d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe0d590 msgr2=0x5618901d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe0d590 0x5618901d0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 msgr2=0x5618901fa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901fa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/3292229557 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618900ec750 con 0x56188fe053b0 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.370+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 0x5618901f6fa0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009bb0 tx=0x7fc19c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7411eaa8b878a12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc193fff640 1 -- 172.21.15.146:0/3292229557 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c036040 con 0x56188fe053b0 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc193fff640 1 -- 172.21.15.146:0/3292229557 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c042070 con 0x56188fe053b0 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc193fff640 1 -- 172.21.15.146:0/3292229557 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c004500 con 0x56188fe053b0 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 -- 172.21.15.146:0/3292229557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 msgr2=0x5618901f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 0x5618901f6fa0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009bb0 tx=0x7fc19c02e000 comp rx=0 tx=0).stop 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 -- 172.21.15.146:0/3292229557 shutdown_connections 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe0d590 0x5618901d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe053b0 0x5618901f6fa0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/3292229557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.371+0000 7fc1bfe18740 1 -- 172.21.15.146:0/3292229557 >> 172.21.15.146:0/3292229557 conn(0x56188fe223c0 msgr2=0x5618901cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.372+0000 7fc1bfe18740 1 -- 172.21.15.146:0/3292229557 shutdown_connections 2024-04-25T15:21:18.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.372+0000 7fc1bfe18740 1 -- 172.21.15.146:0/3292229557 wait complete. 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.372+0000 7fc1bfe18740 1 -- start start 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.372+0000 7fc1bfe18740 1 Processor -- start 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x561890212a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5618900bbfe0 con 0x56188fe0d590 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561890225d90 con 0x56188fe2e270 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x561890213080 con 0x56188fe053b0 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1bfe18740 10 client.0 _ll_register_callbacks cb 0x5618900d2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38256/0 (socket says 172.21.15.146:38256) 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/4090877846 learned_addr learned my addr 172.21.15.146:0/4090877846 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 msgr2=0x561890212a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1b259c640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x561890212a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x561890212a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 msgr2=0x5618901f94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901f94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.374+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/4090877846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56188fe462d0 con 0x56188fe0d590 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214fe0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc19c03f040 tx=0x7fc19c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384031ca1c097f63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1917fa640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1917fa640 1 -- 172.21.15.146:0/4090877846 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c003a20 con 0x56188fe0d590 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc19c042070 con 0x56188fe0d590 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1917fa640 1 -- 172.21.15.146:0/4090877846 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc19c003bc0 con 0x56188fe0d590 2024-04-25T15:21:18.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1917fa640 1 -- 172.21.15.146:0/4090877846 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fc19c059050 con 0x56188fe0d590 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 10 client.5745 fetch_fsmap learned FSMap version 17 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x56188ff106b0 con 0x56188fe0d590 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1917fa640 1 -- 172.21.15.146:0/4090877846 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c042070 con 0x56188fe0d590 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1917fa640 1 -- 172.21.15.146:0/4090877846 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7fc19c05c020 con 0x56188fe0d590 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 10 client.5745 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 10 client.5745 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 20 client.5745 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56188fdece40 0x561890224c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 4 client.5745 mds_command: new command op to 5556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 --> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561890142350 con 0x56188fdece40 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.375+0000 7fc1b259c640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56188fdece40 0x561890224c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.376+0000 7fc1917fa640 10 client.5745 ms_handle_connect on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:18.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.376+0000 7fc1b259c640 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56188fdece40 0x561890224c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2527593b6009a576 server_cookie=252f3d11ebeb173 in_seq=0 out_seq=0 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1917fa640 1 -- 172.21.15.146:0/4090877846 <== mds.0 v2:172.21.15.177:6833/539736758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561890142350 con 0x56188fdece40 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1917fa640 10 client.5745 handle_command_reply: tid=0 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 client.5745 shutdown 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 msgr2=0x561890214fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214fe0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc19c03f040 tx=0x7fc19c02c000 comp rx=0 tx=0).stop 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 shutdown_connections 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56188fdece40 0x561890224c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1917fa640 0 client.5745 ms_handle_reset on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214fe0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4090877846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x561890212a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 >> 172.21.15.146:0/4090877846 conn(0x56188fe223c0 msgr2=0x5618901cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 shutdown_connections 2024-04-25T15:21:18.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4090877846 wait complete. 2024-04-25T15:21:18.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.377+0000 7fc1bfe18740 20 client.5745 trim_cache size 0 max 16384 2024-04-25T15:21:18.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.396+0000 7fc1bfe18740 1 Processor -- start 2024-04-25T15:21:18.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.396+0000 7fc1bfe18740 1 -- start start 2024-04-25T15:21:18.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618901fad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe0d590 0x561890213fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 0x5618901fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4202bb0 con 0x56188fe2e270 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4201f10 con 0x56188fe0d590 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b4202560 con 0x56188fe053b0 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1b2d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 0x5618901fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1b2d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 0x5618901fb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38272/0 (socket says 172.21.15.146:38272) 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.397+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/4004911473 learned_addr learned my addr 172.21.15.146:0/4004911473 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/4004911473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 msgr2=0x5618901fad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618901fad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/4004911473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe0d590 msgr2=0x561890213fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe0d590 0x561890213fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1b2d9d640 1 -- 172.21.15.146:0/4004911473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618901cb1b0 con 0x56188fe2e270 2024-04-25T15:21:18.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 0x5618901fb2a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac05e600 tx=0x7fc1ac012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bbca4d16cf3ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1937fe640 1 -- 172.21.15.146:0/4004911473 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac05e410 con 0x56188fe2e270 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1937fe640 1 -- 172.21.15.146:0/4004911473 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1ac05f000 con 0x56188fe2e270 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.398+0000 7fc1937fe640 1 -- 172.21.15.146:0/4004911473 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac05e900 con 0x56188fe2e270 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4004911473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 msgr2=0x5618901fb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 0x5618901fb2a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac05e600 tx=0x7fc1ac012000 comp rx=0 tx=0).stop 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4004911473 shutdown_connections 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe2e270 0x5618901fb2a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe0d590 0x561890213fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/4004911473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618901fad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4004911473 >> 172.21.15.146:0/4004911473 conn(0x56188fe223c0 msgr2=0x5618901fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4004911473 shutdown_connections 2024-04-25T15:21:18.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.399+0000 7fc1bfe18740 1 -- 172.21.15.146:0/4004911473 wait complete. 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.400+0000 7fc1bfe18740 1 -- start start 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.400+0000 7fc1bfe18740 1 Processor -- start 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618902784d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5618901fc2f0 con 0x56188fe0d590 2024-04-25T15:21:18.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5618901f9a60 con 0x56188fe2e270 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5618901fc940 con 0x56188fe053b0 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1bfe18740 10 client.0 _ll_register_callbacks cb 0x7fc1a00036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1a9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1a9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38278/0 (socket says 172.21.15.146:38278) 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/1400017925 learned_addr learned my addr 172.21.15.146:0/1400017925 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1b259c640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618902784d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.401+0000 7fc1b2d9d640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 msgr2=0x5618902784d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618902784d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 msgr2=0x5618901cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1a9d9b640 1 -- 172.21.15.146:0/1400017925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561890214980 con 0x56188fe0d590 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1b259c640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618902784d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:18.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1a9d9b640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214440 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc19c02ad30 tx=0x7fc19c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0d2a38046d73d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1917fa640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1917fa640 1 -- 172.21.15.146:0/1400017925 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c036070 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1917fa640 1 -- 172.21.15.146:0/1400017925 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc19c05ec30 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1ac05f000 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1917fa640 1 -- 172.21.15.146:0/1400017925 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fc19c065050 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 10 client.5751 fetch_fsmap learned FSMap version 17 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5618901d0190 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1917fa640 1 -- 172.21.15.146:0/1400017925 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1ac05f000 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1917fa640 1 -- 172.21.15.146:0/1400017925 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7fc19c068020 con 0x56188fe0d590 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 10 client.5751 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 10 client.5751 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 20 client.5751 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x561890223450 0x561890223860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 4 client.5751 mds_command: new command op to 5556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.402+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 --> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] -- command(tid 0: {"prefix": "status"}) -- 0x56189013c1c0 con 0x561890223450 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1b259c640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x561890223450 0x561890223860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1b259c640 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x561890223450 0x561890223860 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1445c736111c8e0e server_cookie=90a57b29fdde780e in_seq=0 out_seq=0 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1917fa640 10 client.5751 ms_handle_connect on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1917fa640 1 -- 172.21.15.146:0/1400017925 <== mds.0 v2:172.21.15.177:6833/539736758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x56189013c1c0 con 0x561890223450 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1917fa640 10 client.5751 handle_command_reply: tid=0 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 client.5751 shutdown 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 msgr2=0x561890214440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214440 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc19c02ad30 tx=0x7fc19c004730 comp rx=0 tx=0).stop 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 shutdown_connections 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x561890223450 0x561890223860 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56188fe2e270 0x5618901cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56188fe0d590 0x561890214440 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1917fa640 0 client.5751 ms_handle_reset on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.403+0000 7fc1bfe18740 1 --2- 172.21.15.146:0/1400017925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56188fe053b0 0x5618902784d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 >> 172.21.15.146:0/1400017925 conn(0x56188fe223c0 msgr2=0x5618901ca850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 shutdown_connections 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1bfe18740 1 -- 172.21.15.146:0/1400017925 wait complete. 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1bfe18740 20 client.5751 trim_cache size 0 max 16384 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "c6e9a678-3722-48f4-a297-70f509c13e29", 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5556, 2024-04-25T15:21:18.410 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 10.127145595, 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 17, 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 39, 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 39, 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 17.004556391000001 2024-04-25T15:21:18.411 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc1940744b0 msgr2=0x7fc1940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc1940744b0 0x7fc1940769c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4000c00 tx=0x7fc1a4009040 comp rx=0 tx=0).stop 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 msgr2=0x7fc1b40cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 0x7fc1b40cff30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40726b0 tx=0x7fc1ac00ead0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 shutdown_connections 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc1940744b0 0x7fc1940769c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1b40cfb20 0x7fc1b40cff30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1b4153e80 0x7fc1b40cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 --2- 172.21.15.146:0/2253557601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1b4155690 0x7fc1b4156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 >> 172.21.15.146:0/2253557601 conn(0x7fc1b409b2b0 msgr2=0x7fc1b40cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 shutdown_connections 2024-04-25T15:21:18.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.404+0000 7fc1b8f6b640 1 -- 172.21.15.146:0/2253557601 wait complete. 2024-04-25T15:21:18.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:21:18.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.523+0000 7efc9de42640 1 Processor -- start 2024-04-25T15:21:18.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.523+0000 7efc9de42640 1 -- start start 2024-04-25T15:21:18.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc9de42640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc980d0730 0x7efc980d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc9de42640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efc9806ccc0 0x7efc9806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc9de42640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc980b9200 con 0x7efc980d4f60 2024-04-25T15:21:18.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc9de42640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7efc9806c550 con 0x7efc980d0730 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc9de42640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7efc9806c830 con 0x7efc9806ccc0 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc96ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc980d0730 0x7efc980d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc96ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc980d0730 0x7efc980d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38288/0 (socket says 172.21.15.146:38288) 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc96ffd640 1 -- 172.21.15.146:0/4088387744 learned_addr learned my addr 172.21.15.146:0/4088387744 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.524+0000 7efc97fff640 1 --2- 172.21.15.146:0/4088387744 >> v2:172.21.15.177:3300/0 conn(0x7efc9806ccc0 0x7efc9806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.525+0000 7efc96ffd640 1 -- 172.21.15.146:0/4088387744 >> v2:172.21.15.177:3300/0 conn(0x7efc9806ccc0 msgr2=0x7efc9806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.525+0000 7efc96ffd640 1 --2- 172.21.15.146:0/4088387744 >> v2:172.21.15.177:3300/0 conn(0x7efc9806ccc0 0x7efc9806d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.525+0000 7efc96ffd640 1 -- 172.21.15.146:0/4088387744 >> v1:172.21.15.177:6789/0 conn(0x7efc980d4f60 legacy=0x7efc980d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.525+0000 7efc96ffd640 1 -- 172.21.15.146:0/4088387744 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc980d1070 con 0x7efc980d0730 2024-04-25T15:21:18.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.525+0000 7efc96ffd640 1 --2- 172.21.15.146:0/4088387744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d0730 0x7efc980d5370 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7efc980d3e10 tx=0x7efc8c02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4c34694cda98ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.525+0000 7efc967fc640 1 -- 172.21.15.146:0/4088387744 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c037070 con 0x7efc980d0730 2024-04-25T15:21:18.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc967fc640 1 -- 172.21.15.146:0/4088387744 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc8c032440 con 0x7efc980d0730 2024-04-25T15:21:18.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc967fc640 1 -- 172.21.15.146:0/4088387744 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c032810 con 0x7efc980d0730 2024-04-25T15:21:18.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 -- 172.21.15.146:0/4088387744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d0730 msgr2=0x7efc980d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 --2- 172.21.15.146:0/4088387744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d0730 0x7efc980d5370 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7efc980d3e10 tx=0x7efc8c02a150 comp rx=0 tx=0).stop 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 -- 172.21.15.146:0/4088387744 shutdown_connections 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 --2- 172.21.15.146:0/4088387744 >> v2:172.21.15.177:3300/0 conn(0x7efc9806ccc0 0x7efc9806d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 --2- 172.21.15.146:0/4088387744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d0730 0x7efc980d5370 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 -- 172.21.15.146:0/4088387744 >> 172.21.15.146:0/4088387744 conn(0x7efc980d2450 msgr2=0x7efc980d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 -- 172.21.15.146:0/4088387744 shutdown_connections 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.526+0000 7efc9de42640 1 -- 172.21.15.146:0/4088387744 wait complete. 2024-04-25T15:21:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.528+0000 7efc9de42640 1 Processor -- start 2024-04-25T15:21:18.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.528+0000 7efc9de42640 1 -- start start 2024-04-25T15:21:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.529+0000 7efc9de42640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efc9806ccc0 0x7efc980b9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.529+0000 7efc9de42640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 0x7efc9814dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.529+0000 7efc9de42640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc98161770 0x7efc98161b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.529+0000 7efc9de42640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efc980b5150 con 0x7efc980d4f60 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.529+0000 7efc9de42640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc980d15f0 con 0x7efc98161770 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.529+0000 7efc9de42640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efc980b52d0 con 0x7efc9806ccc0 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 0x7efc9814dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 0x7efc9814dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38302/0 (socket says 172.21.15.146:38302) 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 -- 172.21.15.146:0/3164660476 learned_addr learned my addr 172.21.15.146:0/3164660476 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc977fe640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efc9806ccc0 0x7efc980b9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc97fff640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc98161770 0x7efc98161b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 -- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efc9806ccc0 msgr2=0x7efc980b9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efc9806ccc0 0x7efc980b9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 -- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc98161770 msgr2=0x7efc98161b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc98161770 0x7efc98161b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 -- 172.21.15.146:0/3164660476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc981620c0 con 0x7efc980d4f60 2024-04-25T15:21:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc96ffd640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 0x7efc9814dd60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c041020 con 0x7efc980d4f60 2024-04-25T15:21:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc8c0306c0 con 0x7efc980d4f60 2024-04-25T15:21:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9815d820 con 0x7efc980d4f60 2024-04-25T15:21:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c037040 con 0x7efc980d4f60 2024-04-25T15:21:18.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.530+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9815dde0 con 0x7efc980d4f60 2024-04-25T15:21:18.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.531+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc9806a0e0 con 0x7efc980d4f60 2024-04-25T15:21:18.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.532+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc8c044020 con 0x7efc980d4f60 2024-04-25T15:21:18.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.533+0000 7efc7ffff640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efc840744b0 0x7efc840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:18.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.533+0000 7efc977fe640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efc840744b0 0x7efc840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:18.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.533+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7efc8c0b5b00 con 0x7efc980d4f60 2024-04-25T15:21:18.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.533+0000 7efc977fe640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efc840744b0 0x7efc840769c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efc980b4880 tx=0x7efc88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:18.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.537+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc8c04b030 con 0x7efc980d4f60 2024-04-25T15:21:18.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.752+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efc980b4660 con 0x7efc980d4f60 2024-04-25T15:21:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.752+0000 7efc7ffff640 1 -- 172.21.15.146:0/3164660476 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7efc981620c0 con 0x7efc980d4f60 2024-04-25T15:21:18.754 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:21:18.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efc840744b0 msgr2=0x7efc840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efc840744b0 0x7efc840769c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efc980b4880 tx=0x7efc88009040 comp rx=0 tx=0).stop 2024-04-25T15:21:18.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 msgr2=0x7efc9814dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:18.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 0x7efc9814dd60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000c00 tx=0x7efc8c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 shutdown_connections 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efc840744b0 0x7efc840769c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc98161770 0x7efc98161b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc980d4f60 0x7efc9814dd60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 --2- 172.21.15.146:0/3164660476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efc9806ccc0 0x7efc980b9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 >> 172.21.15.146:0/3164660476 conn(0x7efc980d2450 msgr2=0x7efc98164290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 shutdown_connections 2024-04-25T15:21:18.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18.754+0000 7efc9de42640 1 -- 172.21.15.146:0/3164660476 wait complete. 2024-04-25T15:21:18.767 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:21:18.767 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:21:18.768 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:21:18.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:18.768 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:21:18.783 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:21:18.784 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42682sec preferred_lft 42682sec 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.785 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7820:1ff:feb4:c7f1/64 scope link 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:18.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:18.786 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:21:18.847 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:21:18.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:18.848 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:21:18.904 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:21:18.905 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:21:18.905 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:21:18.905 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:21:18.905 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:21:18.905 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:21:18.905 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:21:18.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:21:18.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:21:18.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:21:18.906 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:21:18.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:21:19.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:19.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.028 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:21:19.043 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:21:19.043 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:21:19.043 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:21:19.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-25T15:21:19.108 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T15:21:19.108 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:19.108 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:19.162 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:21:19.162 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:19.163 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:19.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:21:19.282 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:19.338 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:21:19.338 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:21:19.339 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:21:19.340 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:21:19.341 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:21:19.341 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:19.341 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:19.341 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:19.341 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:19.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.341 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:21:19.393 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:21:19.393 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-25T15:21:19.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:21:19.459 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:19.518 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:21:19.518 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:21:19.518 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:21:19.518 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:21:19.518 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:21:19.519 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:21:19.520 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:21:19.521 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:19.521 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:19.521 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:19.521 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:19.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.521 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:21:19.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:2024-04-25T15:21:19.525+0000 7f386872b480 -1 init, newargv = 0x56331a4ed390 newargc=15 2024-04-25T15:21:19.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[51752]: starting ceph client 2024-04-25T15:21:19.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[51752]: starting fuse 2024-04-25T15:21:19.541 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:21:19.541 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:21:19.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:21:19.541 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:21:19.541 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:21:19.541 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:21:19.541 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:21:19.541 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:21:19.542 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:21:19.543 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:21:19.543 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:21:19.543 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:21:19.543 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:19.543 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.0")) 2024-04-25T15:21:19.543 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:21:19.659 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.0.51752.asok 2024-04-25T15:21:19.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:19.669 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51752.asok 2024-04-25T15:21:19.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51752.asok status 2024-04-25T15:21:19.812 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:21:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:21:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:21:19.812 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "0", 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "51752", 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5763, 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 5763 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:21:19.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3416509291 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3416509291 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.5763 192.168.0.1:0/3416509291", 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/3416509291", 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 17, 2024-04-25T15:21:19.814 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 39, 2024-04-25T15:21:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:21:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:21:19.815 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:21:19.815 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:19.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.821 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-04-25T15:21:19.821 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:19.836 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-04-25T15:21:19.837 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:19.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:19.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:19.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:21:20.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.046+0000 7fc707157640 1 Processor -- start 2024-04-25T15:21:20.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.046+0000 7fc707157640 1 -- start start 2024-04-25T15:21:20.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc707157640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc707157640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7000d0ac0 0x7fc7000d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc707157640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7000d1410 con 0x7fc7000cbc90 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc707157640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc7000cfd10 con 0x7fc7000d0ac0 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc707157640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc7000cf420 con 0x7fc700153ca0 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7000d0ac0 0x7fc7000d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7000d0ac0 0x7fc7000d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38304/0 (socket says 172.21.15.146:38304) 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 -- 172.21.15.146:0/697912303 learned_addr learned my addr 172.21.15.146:0/697912303 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc6fffff640 1 --2- 172.21.15.146:0/697912303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 -- 172.21.15.146:0/697912303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 msgr2=0x7fc70015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 --2- 172.21.15.146:0/697912303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70015dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 -- 172.21.15.146:0/697912303 >> v1:172.21.15.177:6789/0 conn(0x7fc7000cbc90 legacy=0x7fc7000d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.048+0000 7fc7056f4640 1 -- 172.21.15.146:0/697912303 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7000cc270 con 0x7fc7000d0ac0 2024-04-25T15:21:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.049+0000 7fc7056f4640 1 --2- 172.21.15.146:0/697912303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc7000d0ed0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009e50 tx=0x7fc6f802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdde3c5e1b5f16a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.049+0000 7fc6ff7fe640 1 -- 172.21.15.146:0/697912303 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8036070 con 0x7fc7000d0ac0 2024-04-25T15:21:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.049+0000 7fc6ff7fe640 1 -- 172.21.15.146:0/697912303 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f8031600 con 0x7fc7000d0ac0 2024-04-25T15:21:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.049+0000 7fc6ff7fe640 1 -- 172.21.15.146:0/697912303 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80319d0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 -- 172.21.15.146:0/697912303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 msgr2=0x7fc7000d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 --2- 172.21.15.146:0/697912303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc7000d0ed0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8009e50 tx=0x7fc6f802a6c0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 -- 172.21.15.146:0/697912303 shutdown_connections 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 --2- 172.21.15.146:0/697912303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc7000d0ed0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 --2- 172.21.15.146:0/697912303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 -- 172.21.15.146:0/697912303 >> 172.21.15.146:0/697912303 conn(0x7fc7000ce0c0 msgr2=0x7fc7000ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:20.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 -- 172.21.15.146:0/697912303 shutdown_connections 2024-04-25T15:21:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.050+0000 7fc707157640 1 -- 172.21.15.146:0/697912303 wait complete. 2024-04-25T15:21:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.051+0000 7fc707157640 1 Processor -- start 2024-04-25T15:21:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.051+0000 7fc707157640 1 -- start start 2024-04-25T15:21:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc707157640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7000cbc90 0x7fc7000aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc707157640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc700156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc707157640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70006cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc707157640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc70015e1a0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc707157640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7000cf9a0 con 0x7fc7000cbc90 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc707157640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc7001540b0 con 0x7fc700153ca0 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc700156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc700156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38320/0 (socket says 172.21.15.146:38320) 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 -- 172.21.15.146:0/34659177 learned_addr learned my addr 172.21.15.146:0/34659177 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 -- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 msgr2=0x7fc70006cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc704ef3640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7000cbc90 0x7fc7000aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70006cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 -- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7000cbc90 msgr2=0x7fc7000aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7000cbc90 0x7fc7000aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 -- 172.21.15.146:0/34659177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70015a7f0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.053+0000 7fc6fffff640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc700156d10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400c7f0 tx=0x7fc6f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:20.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.054+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4015070 con 0x7fc7000d0ac0 2024-04-25T15:21:20.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.054+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f4004220 con 0x7fc7000d0ac0 2024-04-25T15:21:20.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.054+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7000af6a0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.054+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4007620 con 0x7fc7000d0ac0 2024-04-25T15:21:20.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.054+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7000afc60 con 0x7fc7000d0ac0 2024-04-25T15:21:20.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.055+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70006a0e0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.055+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc6f40079d0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.056+0000 7fc6fd7fa640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc6ec0744b0 0x7fc6ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:20.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.056+0000 7fc704ef3640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc6ec0744b0 0x7fc6ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:20.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.056+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc6f4095450 con 0x7fc7000d0ac0 2024-04-25T15:21:20.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.056+0000 7fc704ef3640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc6ec0744b0 0x7fc6ec0769c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc70006c600 tx=0x7fc6f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:20.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.061+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc70015a7f0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.274+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc7000711c0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.275+0000 7fc6fd7fa640 1 -- 172.21.15.146:0/34659177 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fc6f405f9d0 con 0x7fc7000d0ac0 2024-04-25T15:21:20.277 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc6ec0744b0 msgr2=0x7fc6ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc6ec0744b0 0x7fc6ec0769c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc70006c600 tx=0x7fc6f0008040 comp rx=0 tx=0).stop 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 msgr2=0x7fc700156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc700156d10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400c7f0 tx=0x7fc6f400ccb0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 shutdown_connections 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc6ec0744b0 0x7fc6ec0769c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc700153ca0 0x7fc70006cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7000d0ac0 0x7fc700156d10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 --2- 172.21.15.146:0/34659177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7000cbc90 0x7fc7000aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:20.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 >> 172.21.15.146:0/34659177 conn(0x7fc7000ce0c0 msgr2=0x7fc7000b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:20.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 shutdown_connections 2024-04-25T15:21:20.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:20.277+0000 7fc707157640 1 -- 172.21.15.146:0/34659177 wait complete. 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:21:20.290 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:21:20.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:20.290 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:20.310 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42654sec preferred_lft 42654sec 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.311 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::7cc7:d2ff:fef5:71b6/64 scope link 2024-04-25T15:21:20.312 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:20.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:20.313 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:21:20.379 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:21:20.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:20.381 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:21:20.439 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:21:20.440 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:21:20.440 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:21:20.440 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:21:20.440 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:21:20.440 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:21:20.440 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:21:20.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:21:20.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:21:20.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:21:20.441 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:21:20.534 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:21:20.566 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:20.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:20.572 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:21:20.587 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:21:20.588 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:21:20.588 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:21:20.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:20.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-04-25T15:21:20.653 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-25T15:21:20.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:20.653 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:20.708 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:21:20.709 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:20.709 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:20.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:21:20.828 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:20.884 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:21:20.884 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:21:20.884 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:21:20.884 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:21:20.884 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:21:20.885 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:21:20.886 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:21:20.887 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:21:20.887 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:20.887 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:20.887 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:20.887 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:20.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:20.887 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:21:20.939 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:21:20.939 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-25T15:21:20.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:21:21.005 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:21.062 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:21:21.063 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:21.064 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:21.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:21.066 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:21:21.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:2024-04-25T15:21:21.069+0000 7fa4e098d480 -1 init, newargv = 0x7fa4c4005eb0 newargc=15 2024-04-25T15:21:21.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[42086]: starting ceph client 2024-04-25T15:21:21.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[42086]: starting fuse 2024-04-25T15:21:21.085 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:21:21.085 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:21:21.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:21:21.085 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:21:21.085 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:21:21.085 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:21:21.085 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:21:21.085 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:21:21.086 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.1")) 2024-04-25T15:21:21.087 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:21:21.201 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.1.42086.asok 2024-04-25T15:21:21.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:21.211 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42086.asok 2024-04-25T15:21:21.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:21.211 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42086.asok status 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "1", 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:21:21.352 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "42086", 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/" 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 5467, 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 5467 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:21:21.353 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 917963072 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 917963072 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.5467 192.168.0.1:0/917963072", 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.1:0/917963072", 2024-04-25T15:21:21.354 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 1, 2024-04-25T15:21:21.355 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 17, 2024-04-25T15:21:21.355 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 39, 2024-04-25T15:21:21.355 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:21:21.355 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:21:21.355 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:21:21.355 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:21:21.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:21.361 DEBUG:teuthology.orchestra.run.smithi177:> stat --file-system '--printf=%T 2024-04-25T15:21:21.361 DEBUG:teuthology.orchestra.run.smithi177:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:21.376 INFO:teuthology.orchestra.run.smithi177.stdout:fuseblk 2024-04-25T15:21:21.376 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:21.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:21.377 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:21.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 25 2024-04-25T15:21:21.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.562+0000 7f5c6c292640 1 Processor -- start 2024-04-25T15:21:21.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.562+0000 7f5c6c292640 1 -- start start 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.563+0000 7f5c6c292640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5c6406dc70 0x7f5c64161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.563+0000 7f5c6c292640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c64157870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.563+0000 7f5c6c292640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6415e450 con 0x7f5c64070b20 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.563+0000 7f5c6c292640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f5c6415ed80 con 0x7f5c6406dc70 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.563+0000 7f5c6c292640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6415e1f0 con 0x7f5c64162180 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6a82f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c64157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6982d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5c6406dc70 0x7f5c64161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6a02e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5c64070b20 0x7f5c6406d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:37252/0 (socket says 172.21.15.146:37252) 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6a02e640 1 -- 172.21.15.146:0/2084648676 learned_addr learned my addr 172.21.15.146:0/2084648676 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6982d640 1 -- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 msgr2=0x7f5c64157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:21.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6982d640 1 --2- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c64157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6982d640 1 -- 172.21.15.146:0/2084648676 >> v1:172.21.15.177:6789/0 conn(0x7f5c64070b20 legacy=0x7f5c6406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6902c640 1 -- 172.21.15.146:0/2084648676 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883196338 0 0) 0x7f5c6415e450 con 0x7f5c64070b20 2024-04-25T15:21:21.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6982d640 1 -- 172.21.15.146:0/2084648676 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6415e730 con 0x7f5c6406dc70 2024-04-25T15:21:21.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.564+0000 7f5c6a82f640 1 --2- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c64157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.565+0000 7f5c6982d640 1 --2- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c6406dc70 0x7f5c64161c40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c540097a0 tx=0x7f5c5402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c1042d18ee5358e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.565+0000 7f5c6902c640 1 -- 172.21.15.146:0/2084648676 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54037070 con 0x7f5c6406dc70 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.565+0000 7f5c6902c640 1 -- 172.21.15.146:0/2084648676 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c5402ae50 con 0x7f5c6406dc70 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.565+0000 7f5c6902c640 1 -- 172.21.15.146:0/2084648676 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54032660 con 0x7f5c6406dc70 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 -- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c6406dc70 msgr2=0x7f5c64161c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c6406dc70 0x7f5c64161c40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c540097a0 tx=0x7f5c5402a100 comp rx=0 tx=0).stop 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 -- 172.21.15.146:0/2084648676 shutdown_connections 2024-04-25T15:21:21.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c64157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2084648676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c6406dc70 0x7f5c64161c40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 -- 172.21.15.146:0/2084648676 >> 172.21.15.146:0/2084648676 conn(0x7f5c640d1c10 msgr2=0x7f5c640733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 -- 172.21.15.146:0/2084648676 shutdown_connections 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.566+0000 7f5c6c292640 1 -- 172.21.15.146:0/2084648676 wait complete. 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.567+0000 7f5c6c292640 1 Processor -- start 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.568+0000 7f5c6c292640 1 -- start start 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6c292640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 0x7f5c64150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6c292640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c6409b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6c292640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c640cf5b0 0x7f5c640cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6c292640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64070f30 con 0x7f5c64070b20 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6c292640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64162590 con 0x7f5c640cf5b0 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6c292640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5c640710b0 con 0x7f5c64162180 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 0x7f5c64150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 0x7f5c64150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56868/0 (socket says 172.21.15.146:56868) 2024-04-25T15:21:21.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 -- 172.21.15.146:0/2107151304 learned_addr learned my addr 172.21.15.146:0/2107151304 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6982d640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c6409b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a82f640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c640cf5b0 0x7f5c640cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 -- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 msgr2=0x7f5c6409b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c6409b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 -- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c640cf5b0 msgr2=0x7f5c640cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c640cf5b0 0x7f5c640cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.569+0000 7f5c6a02e640 1 -- 172.21.15.146:0/2107151304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6415b460 con 0x7f5c64070b20 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.570+0000 7f5c6a02e640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 0x7f5c64150ec0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c460 tx=0x7f5c6000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.570+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60018070 con 0x7f5c64070b20 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.570+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c60007500 con 0x7f5c64070b20 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.570+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c640cff00 con 0x7f5c64070b20 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.570+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60004a20 con 0x7f5c64070b20 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.570+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6409c170 con 0x7f5c64070b20 2024-04-25T15:21:21.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.571+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6406a0e0 con 0x7f5c64070b20 2024-04-25T15:21:21.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.571+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5c6001b070 con 0x7f5c64070b20 2024-04-25T15:21:21.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.572+0000 7f5c5affd640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c480744e0 0x7f5c480769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.572+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f5c60094550 con 0x7f5c64070b20 2024-04-25T15:21:21.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.572+0000 7f5c6982d640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c480744e0 0x7f5c480769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.572+0000 7f5c6982d640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c480744e0 0x7f5c480769f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c54034000 tx=0x7f5c5402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:21.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.577+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c640cff00 con 0x7f5c64070b20 2024-04-25T15:21:21.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.752+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f5c640af910 con 0x7f5c64070b20 2024-04-25T15:21:21.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.756+0000 7f5c5affd640 1 -- 172.21.15.146:0/2107151304 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7f5c6415b460 con 0x7f5c64070b20 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c480744e0 msgr2=0x7f5c480769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c480744e0 0x7f5c480769f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c54034000 tx=0x7f5c5402e000 comp rx=0 tx=0).stop 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 msgr2=0x7f5c64150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 0x7f5c64150ec0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c460 tx=0x7f5c6000c920 comp rx=0 tx=0).stop 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 shutdown_connections 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c480744e0 0x7f5c480769f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c640cf5b0 0x7f5c640cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c64162180 0x7f5c6409b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 --2- 172.21.15.146:0/2107151304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c64070b20 0x7f5c64150ec0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 >> 172.21.15.146:0/2107151304 conn(0x7f5c640d1c10 msgr2=0x7f5c640d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 shutdown_connections 2024-04-25T15:21:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.758+0000 7f5c6c292640 1 -- 172.21.15.146:0/2107151304 wait complete. 2024-04-25T15:21:21.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_cache_quiesce_splitauth true 2024-04-25T15:21:21.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.883+0000 7f598f8e0640 1 Processor -- start 2024-04-25T15:21:21.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.883+0000 7f598f8e0640 1 -- start start 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.884+0000 7f598f8e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f598815e0f0 0x7f59880d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.884+0000 7f598f8e0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59880d2370 0x7f59880afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.884+0000 7f598f8e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880d2780 con 0x7f59880b0fa0 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.884+0000 7f598f8e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f59880b13b0 con 0x7f598815e0f0 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.884+0000 7f598f8e0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f59880aee10 con 0x7f59880d2370 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598d67c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f59880b0fa0 0x7f59880b3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46312/0 (socket says 172.21.15.146:46312) 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598ce7b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f598815e0f0 0x7f59880d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598d67c640 1 -- 172.21.15.146:0/3982345124 learned_addr learned my addr 172.21.15.146:0/3982345124 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598de7d640 1 --2- 172.21.15.146:0/3982345124 >> v2:172.21.15.177:3300/0 conn(0x7f59880d2370 0x7f59880afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f597ffff640 1 -- 172.21.15.146:0/3982345124 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557129388 0 0) 0x7f5984001110 con 0x7f59880b0fa0 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f597ffff640 1 -- 172.21.15.146:0/3982345124 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5974003520 con 0x7f59880b0fa0 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598ce7b640 1 -- 172.21.15.146:0/3982345124 >> v2:172.21.15.177:3300/0 conn(0x7f59880d2370 msgr2=0x7f59880afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598ce7b640 1 --2- 172.21.15.146:0/3982345124 >> v2:172.21.15.177:3300/0 conn(0x7f59880d2370 0x7f59880afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598ce7b640 1 -- 172.21.15.146:0/3982345124 >> v1:172.21.15.146:6789/0 conn(0x7f59880b0fa0 legacy=0x7f59880b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.885+0000 7f598ce7b640 1 -- 172.21.15.146:0/3982345124 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880af0b0 con 0x7f598815e0f0 2024-04-25T15:21:21.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.886+0000 7f598ce7b640 1 --2- 172.21.15.146:0/3982345124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880d1e30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f59880b4e40 tx=0x7f597802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acac0f98f5ba8b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:21.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.886+0000 7f597ffff640 1 -- 172.21.15.146:0/3982345124 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5978037070 con 0x7f598815e0f0 2024-04-25T15:21:21.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.886+0000 7f597ffff640 1 -- 172.21.15.146:0/3982345124 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5978032440 con 0x7f598815e0f0 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.886+0000 7f597ffff640 1 -- 172.21.15.146:0/3982345124 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5978032810 con 0x7f598815e0f0 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 -- 172.21.15.146:0/3982345124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 msgr2=0x7f59880d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3982345124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880d1e30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f59880b4e40 tx=0x7f597802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 -- 172.21.15.146:0/3982345124 shutdown_connections 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3982345124 >> v2:172.21.15.177:3300/0 conn(0x7f59880d2370 0x7f59880afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3982345124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880d1e30 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 -- 172.21.15.146:0/3982345124 >> 172.21.15.146:0/3982345124 conn(0x7f598809b2b0 msgr2=0x7f59880b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 -- 172.21.15.146:0/3982345124 shutdown_connections 2024-04-25T15:21:21.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.887+0000 7f598f8e0640 1 -- 172.21.15.146:0/3982345124 wait complete. 2024-04-25T15:21:21.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.888+0000 7f598f8e0640 1 Processor -- start 2024-04-25T15:21:21.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.888+0000 7f598f8e0640 1 -- start start 2024-04-25T15:21:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59880b0fa0 0x7f598814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59880d2370 0x7f59881611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f59880af950 con 0x7f598815e0f0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59880af670 con 0x7f59880b0fa0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f598815e500 con 0x7f59880d2370 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880b3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56884/0 (socket says 172.21.15.146:56884) 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 -- 172.21.15.146:0/3150352022 learned_addr learned my addr 172.21.15.146:0/3150352022 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598d67c640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59880b0fa0 0x7f598814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598ce7b640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59880d2370 0x7f59881611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 -- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59880d2370 msgr2=0x7f59881611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59880d2370 0x7f59881611f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 -- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59880b0fa0 msgr2=0x7f598814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59880b0fa0 0x7f598814e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 -- 172.21.15.146:0/3150352022 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5988157df0 con 0x7f598815e0f0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598de7d640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880b3bb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5980009cb0 tx=0x7f598000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:21.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980018020 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5980014480 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.890+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59881582f0 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.891+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980014870 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.891+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59881588b0 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.892+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5988158060 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.892+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5980005d10 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.893+0000 7f597dffb640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f596c0744e0 0x7f596c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.893+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f5980096160 con 0x7f598815e0f0 2024-04-25T15:21:21.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.893+0000 7f598d67c640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f596c0744e0 0x7f596c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.893+0000 7f598d67c640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f596c0744e0 0x7f596c0769f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5984007fd0 tx=0x7f5984009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:21.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:21.898+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5980022030 con 0x7f598815e0f0 2024-04-25T15:21:22.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.076+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mds_cache_quiesce_splitauth}] v 0) -- 0x7f598815a050 con 0x7f598815e0f0 2024-04-25T15:21:22.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.079+0000 7f597dffb640 1 -- 172.21.15.146:0/3150352022 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_cache_quiesce_splitauth}]=0 v12)=0 v12) ==== 128+0+0 (secure 0 0 0) 0x7f5980010040 con 0x7f598815e0f0 2024-04-25T15:21:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f596c0744e0 msgr2=0x7f596c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f596c0744e0 0x7f596c0769f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5984007fd0 tx=0x7f5984009040 comp rx=0 tx=0).stop 2024-04-25T15:21:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 msgr2=0x7f59880b3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880b3bb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5980009cb0 tx=0x7f598000c490 comp rx=0 tx=0).stop 2024-04-25T15:21:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 shutdown_connections 2024-04-25T15:21:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f596c0744e0 0x7f596c0769f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f598815e0f0 0x7f59880b3bb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59880d2370 0x7f59881611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 --2- 172.21.15.146:0/3150352022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59880b0fa0 0x7f598814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.081+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 >> 172.21.15.146:0/3150352022 conn(0x7f598809b2b0 msgr2=0x7f598815d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.082+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 shutdown_connections 2024-04-25T15:21:22.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.082+0000 7f598f8e0640 1 -- 172.21.15.146:0/3150352022 wait complete. 2024-04-25T15:21:22.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol_quiesce --mode=777 2024-04-25T15:21:22.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.204+0000 7f928ccd1640 1 Processor -- start 2024-04-25T15:21:22.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.204+0000 7f928ccd1640 1 -- start start 2024-04-25T15:21:22.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f928ccd1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92880d4ad0 0x7f92880b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f928ccd1640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f92880d0690 0x7f92880d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f928ccd1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928806de30 con 0x7f928806d310 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f928ccd1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f928806ce80 con 0x7f92880d4ad0 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f928ccd1640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f92880d5290 con 0x7f92880d0690 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f928659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92880d4ad0 0x7f92880b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f9285d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f928806d310 0x7f928806d720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46316/0 (socket says 172.21.15.146:46316) 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f9285d9b640 1 -- 172.21.15.146:0/2009027471 learned_addr learned my addr 172.21.15.146:0/2009027471 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.205+0000 7f9286d9d640 1 --2- 172.21.15.146:0/2009027471 >> v2:172.21.15.177:3300/0 conn(0x7f92880d0690 0x7f92880d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928659c640 1 -- 172.21.15.146:0/2009027471 >> v2:172.21.15.177:3300/0 conn(0x7f92880d0690 msgr2=0x7f92880d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928659c640 1 --2- 172.21.15.146:0/2009027471 >> v2:172.21.15.177:3300/0 conn(0x7f92880d0690 0x7f92880d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928659c640 1 -- 172.21.15.146:0/2009027471 >> v1:172.21.15.146:6789/0 conn(0x7f928806d310 legacy=0x7f928806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928659c640 1 -- 172.21.15.146:0/2009027471 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928806c550 con 0x7f92880d4ad0 2024-04-25T15:21:22.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928559a640 1 -- 172.21.15.146:0/2009027471 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449860871 0 0) 0x7f928806de30 con 0x7f928806d310 2024-04-25T15:21:22.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928659c640 1 --2- 172.21.15.146:0/2009027471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f92880b8f60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f92880b5460 tx=0x7f927802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdecea11a246b5ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928559a640 1 -- 172.21.15.146:0/2009027471 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278007740 con 0x7f92880d4ad0 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928559a640 1 -- 172.21.15.146:0/2009027471 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92780078e0 con 0x7f92880d4ad0 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.206+0000 7f928559a640 1 -- 172.21.15.146:0/2009027471 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278007cb0 con 0x7f92880d4ad0 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 -- 172.21.15.146:0/2009027471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 msgr2=0x7f92880b8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2009027471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f92880b8f60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f92880b5460 tx=0x7f927802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 -- 172.21.15.146:0/2009027471 shutdown_connections 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2009027471 >> v2:172.21.15.177:3300/0 conn(0x7f92880d0690 0x7f92880d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2009027471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f92880b8f60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 -- 172.21.15.146:0/2009027471 >> 172.21.15.146:0/2009027471 conn(0x7f92880d5a50 msgr2=0x7f92880d5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 -- 172.21.15.146:0/2009027471 shutdown_connections 2024-04-25T15:21:22.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.207+0000 7f928ccd1640 1 -- 172.21.15.146:0/2009027471 wait complete. 2024-04-25T15:21:22.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.209+0000 7f928ccd1640 1 Processor -- start 2024-04-25T15:21:22.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.209+0000 7f928ccd1640 1 -- start start 2024-04-25T15:21:22.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.210+0000 7f928ccd1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928806d310 0x7f92881ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.210+0000 7f928ccd1640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92880d0690 0x7f92881ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928ccd1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f9288151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928ccd1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f92880d4ee0 con 0x7f92880d4ad0 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928ccd1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f928806cb10 con 0x7f928806d310 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928ccd1640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f92881551c0 con 0x7f92880d0690 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f9288151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f9288151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56898/0 (socket says 172.21.15.146:56898) 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 -- 172.21.15.146:0/2165183952 learned_addr learned my addr 172.21.15.146:0/2165183952 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928659c640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928806d310 0x7f92881ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9285d9b640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92880d0690 0x7f92881ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 -- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92880d0690 msgr2=0x7f92881ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92880d0690 0x7f92881ebff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 -- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928806d310 msgr2=0x7f92881ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928806d310 0x7f92881ebab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 -- 172.21.15.146:0/2165183952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92881f6370 con 0x7f92880d4ad0 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9285d9b640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92880d0690 0x7f92881ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9286d9d640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f9288151910 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f928000e7f0 tx=0x7f928000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280017070 con 0x7f92880d4ad0 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928000ee40 con 0x7f92880d4ad0 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9288157870 con 0x7f92880d4ad0 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000cb80 con 0x7f92880d4ad0 2024-04-25T15:21:22.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.211+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9288157e30 con 0x7f92880d4ad0 2024-04-25T15:21:22.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.212+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928806a0e0 con 0x7f92880d4ad0 2024-04-25T15:21:22.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.213+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f928001a070 con 0x7f92880d4ad0 2024-04-25T15:21:22.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.213+0000 7f9276ffd640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92680744b0 0x7f92680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.213+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9280095640 con 0x7f92880d4ad0 2024-04-25T15:21:22.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.213+0000 7f928659c640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92680744b0 0x7f92680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.214+0000 7f928659c640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92680744b0 0x7f92680769c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9278034000 tx=0x7f9278033040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.218+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f928005fc40 con 0x7f92880d4ad0 2024-04-25T15:21:22.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.422+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f9288152ff0 con 0x7f92680744b0 2024-04-25T15:21:22.428 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:21:22.427+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:21:22.428 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:21:22.427+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:21:22.428 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:21:22.427+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:21:22.428 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:21:22.427+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:21:22.428 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:21:22.427+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:21:22.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.466+0000 7f9276ffd640 1 -- 172.21.15.146:0/2165183952 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9288152ff0 con 0x7f92680744b0 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92680744b0 msgr2=0x7f92680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92680744b0 0x7f92680769c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9278034000 tx=0x7f9278033040 comp rx=0 tx=0).stop 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 msgr2=0x7f9288151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f9288151910 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f928000e7f0 tx=0x7f928000ebf0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 shutdown_connections 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92680744b0 0x7f92680769c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92880d4ad0 0x7f9288151910 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92880d0690 0x7f92881ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 --2- 172.21.15.146:0/2165183952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928806d310 0x7f92881ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 >> 172.21.15.146:0/2165183952 conn(0x7f92880d5a50 msgr2=0x7f92880686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:22.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 shutdown_connections 2024-04-25T15:21:22.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.468+0000 7f928ccd1640 1 -- 172.21.15.146:0/2165183952 wait complete. 2024-04-25T15:21:22.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_quiesce 2024-04-25T15:21:22.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.590+0000 7ffbeaf2e640 1 Processor -- start 2024-04-25T15:21:22.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.590+0000 7ffbeaf2e640 1 -- start start 2024-04-25T15:21:22.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.591+0000 7ffbeaf2e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe4070b20 0x7ffbe406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.591+0000 7ffbeaf2e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbe406daa0 0x7ffbe4161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.591+0000 7ffbeaf2e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbe406deb0 con 0x7ffbe4162180 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.591+0000 7ffbeaf2e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffbe415ed80 con 0x7ffbe406daa0 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.591+0000 7ffbeaf2e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffbe415e1f0 con 0x7ffbe4070b20 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbe406daa0 0x7ffbe4161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe94cb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffbe4162180 0x7ffbe4157870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46330/0 (socket says 172.21.15.146:46330) 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe94cb640 1 -- 172.21.15.146:0/3191905415 learned_addr learned my addr 172.21.15.146:0/3191905415 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe8cca640 1 --2- 172.21.15.146:0/3191905415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe4070b20 0x7ffbe406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe3fff640 1 -- 172.21.15.146:0/3191905415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe4070b20 msgr2=0x7ffbe406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe3fff640 1 --2- 172.21.15.146:0/3191905415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe4070b20 0x7ffbe406d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe3fff640 1 -- 172.21.15.146:0/3191905415 >> v1:172.21.15.146:6789/0 conn(0x7ffbe4162180 legacy=0x7ffbe4157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe3fff640 1 -- 172.21.15.146:0/3191905415 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe415e450 con 0x7ffbe406daa0 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.592+0000 7ffbe37fe640 1 -- 172.21.15.146:0/3191905415 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968057477 0 0) 0x7ffbdc001490 con 0x7ffbe4162180 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.593+0000 7ffbe3fff640 1 --2- 172.21.15.146:0/3191905415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe406daa0 0x7ffbe4161c40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00097a0 tx=0x7ffbd002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9435e94670de92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.593+0000 7ffbe37fe640 1 -- 172.21.15.146:0/3191905415 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0037070 con 0x7ffbe406daa0 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.593+0000 7ffbe37fe640 1 -- 172.21.15.146:0/3191905415 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbd002ae50 con 0x7ffbe406daa0 2024-04-25T15:21:22.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.593+0000 7ffbe37fe640 1 -- 172.21.15.146:0/3191905415 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0032660 con 0x7ffbe406daa0 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/3191905415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe406daa0 msgr2=0x7ffbe4161c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/3191905415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe406daa0 0x7ffbe4161c40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00097a0 tx=0x7ffbd002a100 comp rx=0 tx=0).stop 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/3191905415 shutdown_connections 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/3191905415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe406daa0 0x7ffbe4161c40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/3191905415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe4070b20 0x7ffbe406d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/3191905415 >> 172.21.15.146:0/3191905415 conn(0x7ffbe40d1c10 msgr2=0x7ffbe40733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/3191905415 shutdown_connections 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.594+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/3191905415 wait complete. 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.595+0000 7ffbeaf2e640 1 Processor -- start 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.595+0000 7ffbeaf2e640 1 -- start start 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbeaf2e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe406daa0 0x7ffbe41e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbeaf2e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe4070b20 0x7ffbe40ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbeaf2e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 0x7ffbe40cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbeaf2e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4162590 con 0x7ffbe4162180 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbeaf2e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe415ea10 con 0x7ffbe4070b20 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbeaf2e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffbe4070f30 con 0x7ffbe406daa0 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 0x7ffbe40cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 0x7ffbe40cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56938/0 (socket says 172.21.15.146:56938) 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 -- 172.21.15.146:0/928124859 learned_addr learned my addr 172.21.15.146:0/928124859 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe8cca640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe406daa0 0x7ffbe41e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe3fff640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe4070b20 0x7ffbe40ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 -- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe406daa0 msgr2=0x7ffbe41e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe406daa0 0x7ffbe41e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 -- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe4070b20 msgr2=0x7ffbe40ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe4070b20 0x7ffbe40ce9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.597+0000 7ffbe94cb640 1 -- 172.21.15.146:0/928124859 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe40b5280 con 0x7ffbe4162180 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.598+0000 7ffbe94cb640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 0x7ffbe40cef10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c700 tx=0x7ffbdc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.598+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc018070 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.598+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbdc00ce10 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.598+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00a880 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.598+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe40b56f0 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.598+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe40b5cb0 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.599+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe40b5450 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.599+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffbdc01b070 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.600+0000 7ffbe17fa640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffbc80744b0 0x7ffbc80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.600+0000 7ffbe8cca640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffbc80744b0 0x7ffbc80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.600+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ffbdc094900 con 0x7ffbe4162180 2024-04-25T15:21:22.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.600+0000 7ffbe8cca640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffbc80744b0 0x7ffbc80769c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009970 tx=0x7ffbd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.604+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffbe40b5450 con 0x7ffbe4162180 2024-04-25T15:21:22.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.810+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "target": ["mon-mgr", ""]}) -- 0x7ffbe406a0e0 con 0x7ffbc80744b0 2024-04-25T15:21:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.815+0000 7ffbe17fa640 1 -- 172.21.15.146:0/928124859 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+70 (secure 0 0 0) 0x7ffbe406a0e0 con 0x7ffbc80744b0 2024-04-25T15:21:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffbc80744b0 msgr2=0x7ffbc80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffbc80744b0 0x7ffbc80769c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009970 tx=0x7ffbd8008040 comp rx=0 tx=0).stop 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 msgr2=0x7ffbe40cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 0x7ffbe40cef10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c700 tx=0x7ffbdc00cbc0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 shutdown_connections 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffbc80744b0 0x7ffbc80769c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbe4162180 0x7ffbe40cef10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe4070b20 0x7ffbe40ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 --2- 172.21.15.146:0/928124859 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe406daa0 0x7ffbe41e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 >> 172.21.15.146:0/928124859 conn(0x7ffbe40d1c10 msgr2=0x7ffbe40d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:22.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 shutdown_connections 2024-04-25T15:21:22.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.817+0000 7ffbeaf2e640 1 -- 172.21.15.146:0/928124859 wait complete. 2024-04-25T15:21:22.830 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:22.830 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:22.846 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:22.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.992+0000 7f6529f57640 1 Processor -- start 2024-04-25T15:21:22.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.993+0000 7f6529f57640 1 -- start start 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6529f57640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65240d19d0 0x7f65240d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6529f57640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240cce10 0x7f65240cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6529f57640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65240b4a60 con 0x7f65240cfbe0 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6529f57640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f65240b5020 con 0x7f65240d19d0 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6529f57640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f65240b4d40 con 0x7f65240cce10 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65240d19d0 0x7f65240d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65240d19d0 0x7f65240d1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56952/0 (socket says 172.21.15.146:56952) 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 -- 172.21.15.146:0/1685771731 learned_addr learned my addr 172.21.15.146:0/1685771731 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6523fff640 1 --2- 172.21.15.146:0/1685771731 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240cce10 0x7f65240cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 -- 172.21.15.146:0/1685771731 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240cce10 msgr2=0x7f65240cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:22.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 --2- 172.21.15.146:0/1685771731 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240cce10 0x7f65240cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 -- 172.21.15.146:0/1685771731 >> v1:172.21.15.177:6789/0 conn(0x7f65240cfbe0 legacy=0x7f65240cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.994+0000 7f6522ffd640 1 -- 172.21.15.146:0/1685771731 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65240cead0 con 0x7f65240d19d0 2024-04-25T15:21:22.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.995+0000 7f6522ffd640 1 --2- 172.21.15.146:0/1685771731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240d19d0 0x7f65240d1de0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6518009870 tx=0x7f651802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bedb977d8a1bba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:22.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.995+0000 7f65227fc640 1 -- 172.21.15.146:0/1685771731 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518037070 con 0x7f65240d19d0 2024-04-25T15:21:22.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.995+0000 7f65227fc640 1 -- 172.21.15.146:0/1685771731 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6518032440 con 0x7f65240d19d0 2024-04-25T15:21:22.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.995+0000 7f65227fc640 1 -- 172.21.15.146:0/1685771731 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518032810 con 0x7f65240d19d0 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 -- 172.21.15.146:0/1685771731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240d19d0 msgr2=0x7f65240d1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 --2- 172.21.15.146:0/1685771731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240d19d0 0x7f65240d1de0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6518009870 tx=0x7f651802a270 comp rx=0 tx=0).stop 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 -- 172.21.15.146:0/1685771731 shutdown_connections 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 --2- 172.21.15.146:0/1685771731 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240cce10 0x7f65240cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 --2- 172.21.15.146:0/1685771731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240d19d0 0x7f65240d1de0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 -- 172.21.15.146:0/1685771731 >> 172.21.15.146:0/1685771731 conn(0x7f65240b73b0 msgr2=0x7f65240b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:22.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 -- 172.21.15.146:0/1685771731 shutdown_connections 2024-04-25T15:21:23.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.996+0000 7f6529f57640 1 -- 172.21.15.146:0/1685771731 wait complete. 2024-04-25T15:21:23.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.998+0000 7f6529f57640 1 Processor -- start 2024-04-25T15:21:23.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.998+0000 7f6529f57640 1 -- start start 2024-04-25T15:21:23.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6529f57640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65240cce10 0x7f65241596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6529f57640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 0x7f65240b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6529f57640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240d19d0 0x7f652415af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6529f57640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f65240cd760 con 0x7f65240cfbe0 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6529f57640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f65240d2320 con 0x7f65240cce10 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6529f57640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f65240cd8e0 con 0x7f65240d19d0 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6522ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 0x7f65240b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6522ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 0x7f65240b2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56960/0 (socket says 172.21.15.146:56960) 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:22.999+0000 7f6522ffd640 1 -- 172.21.15.146:0/941292000 learned_addr learned my addr 172.21.15.146:0/941292000 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f65237fe640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65240cce10 0x7f65241596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6523fff640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240d19d0 0x7f652415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6522ffd640 1 -- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240d19d0 msgr2=0x7f652415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6522ffd640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240d19d0 0x7f652415af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6522ffd640 1 -- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65240cce10 msgr2=0x7f65241596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6522ffd640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65240cce10 0x7f65241596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6522ffd640 1 -- 172.21.15.146:0/941292000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65240b2a70 con 0x7f65240cfbe0 2024-04-25T15:21:23.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6522ffd640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 0x7f65240b2530 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6518000c00 tx=0x7f65180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518041020 con 0x7f65240cfbe0 2024-04-25T15:21:23.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65180306c0 con 0x7f65240cfbe0 2024-04-25T15:21:23.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652415ce90 con 0x7f65240cfbe0 2024-04-25T15:21:23.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518037040 con 0x7f65240cfbe0 2024-04-25T15:21:23.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.000+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652415aad0 con 0x7f65240cfbe0 2024-04-25T15:21:23.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.001+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f652406a0e0 con 0x7f65240cfbe0 2024-04-25T15:21:23.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.002+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6518044020 con 0x7f65240cfbe0 2024-04-25T15:21:23.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.002+0000 7f650bfff640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65100744b0 0x7f65100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.003+0000 7f65237fe640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65100744b0 0x7f65100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.003+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f65180b5a00 con 0x7f65240cfbe0 2024-04-25T15:21:23.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.003+0000 7f65237fe640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65100744b0 0x7f65100769c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6514007fd0 tx=0x7f6514009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.007+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f651804b030 con 0x7f65240cfbe0 2024-04-25T15:21:23.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.183+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65240cb750 con 0x7f65240cfbe0 2024-04-25T15:21:23.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.185+0000 7f650bfff640 1 -- 172.21.15.146:0/941292000 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16990 (secure 0 0 0) 0x7f651807fa40 con 0x7f65240cfbe0 2024-04-25T15:21:23.187 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:23.187 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":39,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:09.275980+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65100744b0 msgr2=0x7f65100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65100744b0 0x7f65100769c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6514007fd0 tx=0x7f6514009040 comp rx=0 tx=0).stop 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 msgr2=0x7f65240b2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 0x7f65240b2530 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6518000c00 tx=0x7f65180046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 shutdown_connections 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f65100744b0 0x7f65100769c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65240d19d0 0x7f652415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65240cfbe0 0x7f65240b2530 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 --2- 172.21.15.146:0/941292000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65240cce10 0x7f65241596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 >> 172.21.15.146:0/941292000 conn(0x7f65240b73b0 msgr2=0x7f65240b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:23.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 shutdown_connections 2024-04-25T15:21:23.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.187+0000 7f6529f57640 1 -- 172.21.15.146:0/941292000 wait complete. 2024-04-25T15:21:23.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:23.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:23.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.313+0000 7f70485e3640 1 Processor -- start 2024-04-25T15:21:23.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.313+0000 7f70485e3640 1 -- start start 2024-04-25T15:21:23.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.314+0000 7f70485e3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70400cf0f0 0x7f70400d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.314+0000 7f70485e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7040071c70 0x7f7040072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.314+0000 7f70485e3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70400cf500 con 0x7f70400b2cd0 2024-04-25T15:21:23.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.314+0000 7f70485e3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f704008da90 con 0x7f70400cf0f0 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.314+0000 7f70485e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f70400d0070 con 0x7f7040071c70 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.314+0000 7f7045b7e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70400cf0f0 0x7f70400d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70400cf0f0 0x7f70400d2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56974/0 (socket says 172.21.15.146:56974) 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 -- 172.21.15.146:0/1379736906 learned_addr learned my addr 172.21.15.146:0/1379736906 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7046b80640 1 --2- 172.21.15.146:0/1379736906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7040071c70 0x7f7040072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 -- 172.21.15.146:0/1379736906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7040071c70 msgr2=0x7f7040072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 --2- 172.21.15.146:0/1379736906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7040071c70 0x7f7040072080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 -- 172.21.15.146:0/1379736906 >> v1:172.21.15.177:6789/0 conn(0x7f70400b2cd0 legacy=0x7f70400d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 -- 172.21.15.146:0/1379736906 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400cf7d0 con 0x7f70400cf0f0 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.315+0000 7f7045b7e640 1 --2- 172.21.15.146:0/1379736906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70400cf0f0 0x7f70400d2030 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7030009870 tx=0x7f703002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc809173479bfb07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f704537d640 1 -- 172.21.15.146:0/1379736906 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030037070 con 0x7f70400cf0f0 2024-04-25T15:21:23.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f704537d640 1 -- 172.21.15.146:0/1379736906 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7030032440 con 0x7f70400cf0f0 2024-04-25T15:21:23.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f704537d640 1 -- 172.21.15.146:0/1379736906 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030032810 con 0x7f70400cf0f0 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 -- 172.21.15.146:0/1379736906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70400cf0f0 msgr2=0x7f70400d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 --2- 172.21.15.146:0/1379736906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70400cf0f0 0x7f70400d2030 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7030009870 tx=0x7f703002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 -- 172.21.15.146:0/1379736906 shutdown_connections 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 --2- 172.21.15.146:0/1379736906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7040071c70 0x7f7040072080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 --2- 172.21.15.146:0/1379736906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70400cf0f0 0x7f70400d2030 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 -- 172.21.15.146:0/1379736906 >> 172.21.15.146:0/1379736906 conn(0x7f70400b52e0 msgr2=0x7f70400b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:23.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 -- 172.21.15.146:0/1379736906 shutdown_connections 2024-04-25T15:21:23.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.316+0000 7f70485e3640 1 -- 172.21.15.146:0/1379736906 wait complete. 2024-04-25T15:21:23.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.318+0000 7f70485e3640 1 Processor -- start 2024-04-25T15:21:23.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.318+0000 7f70485e3640 1 -- start start 2024-04-25T15:21:23.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.319+0000 7f70485e3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 0x7f70401e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.319+0000 7f70485e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70400b2cd0 0x7f70400cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.319+0000 7f70485e3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70400cf0f0 0x7f704014c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.319+0000 7f70485e3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f70400cfd90 con 0x7f7040071c70 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.319+0000 7f70485e3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70400717e0 con 0x7f70400cf0f0 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.319+0000 7f70485e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f70400d2570 con 0x7f70400b2cd0 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 0x7f70401e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 0x7f70401e68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56976/0 (socket says 172.21.15.146:56976) 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 -- 172.21.15.146:0/3965232040 learned_addr learned my addr 172.21.15.146:0/3965232040 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f7045b7e640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70400b2cd0 0x7f70400cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f7046b80640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70400cf0f0 0x7f704014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 -- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70400b2cd0 msgr2=0x7f70400cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70400b2cd0 0x7f70400cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 -- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70400cf0f0 msgr2=0x7f704014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70400cf0f0 0x7f704014c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 -- 172.21.15.146:0/3965232040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400cd630 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f704637f640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 0x7f70401e68e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f703c00c390 tx=0x7f703c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c0052d0 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f703c005470 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c006a60 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f704014b8c0 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.320+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f704014be80 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.321+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f704006a0e0 con 0x7f7040071c70 2024-04-25T15:21:23.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.322+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f703c003640 con 0x7f7040071c70 2024-04-25T15:21:23.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.322+0000 7f7036ffd640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70240744e0 0x7f70240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.322+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f703c0945b0 con 0x7f7040071c70 2024-04-25T15:21:23.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.322+0000 7f7045b7e640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70240744e0 0x7f70240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.323+0000 7f7045b7e640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70240744e0 0x7f70240769f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7030034000 tx=0x7f703002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.327+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70400cd630 con 0x7f7040071c70 2024-04-25T15:21:23.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.501+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f704008d330 con 0x7f7040071c70 2024-04-25T15:21:23.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.502+0000 7f7036ffd640 1 -- 172.21.15.146:0/3965232040 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f703c05e680 con 0x7f7040071c70 2024-04-25T15:21:23.503 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:23.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.503+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70240744e0 msgr2=0x7f70240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.503+0000 7f70485e3640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70240744e0 0x7f70240769f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7030034000 tx=0x7f703002e000 comp rx=0 tx=0).stop 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.503+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 msgr2=0x7f70401e68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.503+0000 7f70485e3640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 0x7f70401e68e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f703c00c390 tx=0x7f703c00c850 comp rx=0 tx=0).stop 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 shutdown_connections 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70240744e0 0x7f70240769f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70400cf0f0 0x7f704014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70400b2cd0 0x7f70400cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 --2- 172.21.15.146:0/3965232040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7040071c70 0x7f70401e68e0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 >> 172.21.15.146:0/3965232040 conn(0x7f70400b52e0 msgr2=0x7f70400b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:23.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 shutdown_connections 2024-04-25T15:21:23.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.504+0000 7f70485e3640 1 -- 172.21.15.146:0/3965232040 wait complete. 2024-04-25T15:21:23.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:23.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.625+0000 7f09ca7b2640 1 Processor -- start 2024-04-25T15:21:23.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.625+0000 7f09ca7b2640 1 -- start start 2024-04-25T15:21:23.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09ca7b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09c40ccd40 0x7f09c409bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09ca7b2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f09c40b2400 0x7f09c40b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09ca7b2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c40b1960 con 0x7f09c40d2680 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09ca7b2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f09c40b3ce0 con 0x7f09c40ccd40 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09ca7b2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f09c40b1c00 con 0x7f09c40b2400 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09c40ccd40 0x7f09c409bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f09c40d2680 0x7f09c40d2a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46346/0 (socket says 172.21.15.146:46346) 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c37fe640 1 -- 172.21.15.146:0/635661420 learned_addr learned my addr 172.21.15.146:0/635661420 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c2ffd640 1 -- 172.21.15.146:0/635661420 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 960991669 0 0) 0x7f09c40b1960 con 0x7f09c40d2680 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c2ffd640 1 -- 172.21.15.146:0/635661420 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09ac003520 con 0x7f09c40d2680 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c8d4f640 1 --2- 172.21.15.146:0/635661420 >> v2:172.21.15.177:3300/0 conn(0x7f09c40b2400 0x7f09c40b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c3fff640 1 -- 172.21.15.146:0/635661420 >> v2:172.21.15.177:3300/0 conn(0x7f09c40b2400 msgr2=0x7f09c40b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c3fff640 1 --2- 172.21.15.146:0/635661420 >> v2:172.21.15.177:3300/0 conn(0x7f09c40b2400 0x7f09c40b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c3fff640 1 -- 172.21.15.146:0/635661420 >> v1:172.21.15.146:6789/0 conn(0x7f09c40d2680 legacy=0x7f09c40d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c3fff640 1 -- 172.21.15.146:0/635661420 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c40b1ee0 con 0x7f09c40ccd40 2024-04-25T15:21:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.627+0000 7f09c8d4f640 1 --2- 172.21.15.146:0/635661420 >> v2:172.21.15.177:3300/0 conn(0x7f09c40b2400 0x7f09c40b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.628+0000 7f09c3fff640 1 --2- 172.21.15.146:0/635661420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40ccd40 0x7f09c409bba0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f09c40742e0 tx=0x7f09b402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a9d9db9795fd80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.628+0000 7f09c2ffd640 1 -- 172.21.15.146:0/635661420 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4037070 con 0x7f09c40ccd40 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.628+0000 7f09c2ffd640 1 -- 172.21.15.146:0/635661420 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b4032440 con 0x7f09c40ccd40 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.628+0000 7f09c2ffd640 1 -- 172.21.15.146:0/635661420 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4032810 con 0x7f09c40ccd40 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 -- 172.21.15.146:0/635661420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40ccd40 msgr2=0x7f09c409bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/635661420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40ccd40 0x7f09c409bba0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f09c40742e0 tx=0x7f09b402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 -- 172.21.15.146:0/635661420 shutdown_connections 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/635661420 >> v2:172.21.15.177:3300/0 conn(0x7f09c40b2400 0x7f09c40b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/635661420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40ccd40 0x7f09c409bba0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 -- 172.21.15.146:0/635661420 >> 172.21.15.146:0/635661420 conn(0x7f09c40b7400 msgr2=0x7f09c40b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 -- 172.21.15.146:0/635661420 shutdown_connections 2024-04-25T15:21:23.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.629+0000 7f09ca7b2640 1 -- 172.21.15.146:0/635661420 wait complete. 2024-04-25T15:21:23.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.631+0000 7f09ca7b2640 1 Processor -- start 2024-04-25T15:21:23.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.631+0000 7f09ca7b2640 1 -- start start 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09ca7b2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 0x7f09c41f91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09ca7b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c40ccd40 0x7f09c4160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09ca7b2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09c40d2680 0x7f09c414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09ca7b2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40b3ac0 con 0x7f09c40b2400 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09ca7b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40b3940 con 0x7f09c40ccd40 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09ca7b2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f09c409b440 con 0x7f09c40d2680 2024-04-25T15:21:23.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09c3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 0x7f09c41f91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09c3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 0x7f09c41f91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56992/0 (socket says 172.21.15.146:56992) 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.632+0000 7f09c3fff640 1 -- 172.21.15.146:0/618530549 learned_addr learned my addr 172.21.15.146:0/618530549 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c37fe640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c40ccd40 0x7f09c4160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c3fff640 1 -- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09c40d2680 msgr2=0x7f09c414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c3fff640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09c40d2680 0x7f09c414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c3fff640 1 -- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c40ccd40 msgr2=0x7f09c4160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c3fff640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c40ccd40 0x7f09c4160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c3fff640 1 -- 172.21.15.146:0/618530549 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c4161a10 con 0x7f09c40b2400 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c37fe640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c40ccd40 0x7f09c4160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c3fff640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 0x7f09c41f91a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f09b402a790 tx=0x7f09b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4037040 con 0x7f09c40b2400 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b4042070 con 0x7f09c40b2400 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c4161cf0 con 0x7f09c40b2400 2024-04-25T15:21:23.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4030960 con 0x7f09c40b2400 2024-04-25T15:21:23.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.633+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c414cda0 con 0x7f09c40b2400 2024-04-25T15:21:23.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.634+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c414c8b0 con 0x7f09c40b2400 2024-04-25T15:21:23.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.634+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f09b4030b00 con 0x7f09c40b2400 2024-04-25T15:21:23.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.635+0000 7f09c0ff9640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09b00744e0 0x7f09b00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.635+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f09b40b5c80 con 0x7f09c40b2400 2024-04-25T15:21:23.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.635+0000 7f09c37fe640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09b00744e0 0x7f09b00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.636+0000 7f09c37fe640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09b00744e0 0x7f09b00769f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f09c415f960 tx=0x7f09b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.640+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09b4080310 con 0x7f09c40b2400 2024-04-25T15:21:23.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.816+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f09c406a0e0 con 0x7f09c40b2400 2024-04-25T15:21:23.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.816+0000 7f09c0ff9640 1 -- 172.21.15.146:0/618530549 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f09b407fc10 con 0x7f09c40b2400 2024-04-25T15:21:23.819 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:23.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.818+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09b00744e0 msgr2=0x7f09b00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.818+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09b00744e0 0x7f09b00769f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f09c415f960 tx=0x7f09b8009040 comp rx=0 tx=0).stop 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.818+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 msgr2=0x7f09c41f91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.818+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 0x7f09c41f91a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f09b402a790 tx=0x7f09b40046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 shutdown_connections 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f09b00744e0 0x7f09b00769f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09c40d2680 0x7f09c414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09c40ccd40 0x7f09c4160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 --2- 172.21.15.146:0/618530549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09c40b2400 0x7f09c41f91a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 >> 172.21.15.146:0/618530549 conn(0x7f09c40b7400 msgr2=0x7f09c40b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 shutdown_connections 2024-04-25T15:21:23.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.819+0000 7f09ca7b2640 1 -- 172.21.15.146:0/618530549 wait complete. 2024-04-25T15:21:23.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:23.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.944+0000 7f7038051640 1 Processor -- start 2024-04-25T15:21:23.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.944+0000 7f7038051640 1 -- start start 2024-04-25T15:21:23.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.945+0000 7f7038051640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70300cf0f0 0x7f70300d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.945+0000 7f7038051640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7030071c70 0x7f7030072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.945+0000 7f7038051640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70300cf500 con 0x7f70300b2cd0 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.945+0000 7f7038051640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f703008da90 con 0x7f70300cf0f0 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.945+0000 7f7038051640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f70300d0070 con 0x7f7030071c70 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.945+0000 7f70355ec640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70300cf0f0 0x7f70300d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70300cf0f0 0x7f70300d2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57008/0 (socket says 172.21.15.146:57008) 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 -- 172.21.15.146:0/3398088019 learned_addr learned my addr 172.21.15.146:0/3398088019 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 -- 172.21.15.146:0/3398088019 >> v2:172.21.15.177:3300/0 conn(0x7f7030071c70 msgr2=0x7f7030072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 --2- 172.21.15.146:0/3398088019 >> v2:172.21.15.177:3300/0 conn(0x7f7030071c70 0x7f7030072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 -- 172.21.15.146:0/3398088019 >> v1:172.21.15.177:6789/0 conn(0x7f70300b2cd0 legacy=0x7f70300d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:23.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 -- 172.21.15.146:0/3398088019 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70300cf7d0 con 0x7f70300cf0f0 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.946+0000 7f70355ec640 1 --2- 172.21.15.146:0/3398088019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300cf0f0 0x7f70300d2030 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f70300b9870 tx=0x7f702002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea2689a167d132 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7034deb640 1 -- 172.21.15.146:0/3398088019 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020037070 con 0x7f70300cf0f0 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7034deb640 1 -- 172.21.15.146:0/3398088019 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7020032440 con 0x7f70300cf0f0 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7034deb640 1 -- 172.21.15.146:0/3398088019 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020032810 con 0x7f70300cf0f0 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 -- 172.21.15.146:0/3398088019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300cf0f0 msgr2=0x7f70300d2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 --2- 172.21.15.146:0/3398088019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300cf0f0 0x7f70300d2030 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f70300b9870 tx=0x7f702002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 -- 172.21.15.146:0/3398088019 shutdown_connections 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 --2- 172.21.15.146:0/3398088019 >> v2:172.21.15.177:3300/0 conn(0x7f7030071c70 0x7f7030072080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 --2- 172.21.15.146:0/3398088019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300cf0f0 0x7f70300d2030 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 -- 172.21.15.146:0/3398088019 >> 172.21.15.146:0/3398088019 conn(0x7f70300b52e0 msgr2=0x7f70300b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 -- 172.21.15.146:0/3398088019 shutdown_connections 2024-04-25T15:21:23.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.947+0000 7f7038051640 1 -- 172.21.15.146:0/3398088019 wait complete. 2024-04-25T15:21:23.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.949+0000 7f7038051640 1 Processor -- start 2024-04-25T15:21:23.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.949+0000 7f7038051640 1 -- start start 2024-04-25T15:21:23.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.950+0000 7f7038051640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7030071c70 0x7f703006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.950+0000 7f7038051640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 0x7f70300cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7038051640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70300cf0f0 0x7f703015c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7038051640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f70300cfd90 con 0x7f70300b2cd0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7038051640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70300717e0 con 0x7f7030071c70 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7038051640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f70300d2570 con 0x7f70300cf0f0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 0x7f70300cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 0x7f70300cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57012/0 (socket says 172.21.15.146:57012) 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 -- 172.21.15.146:0/1954092562 learned_addr learned my addr 172.21.15.146:0/1954092562 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7035ded640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7030071c70 0x7f703006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70365ee640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70300cf0f0 0x7f703015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 -- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70300cf0f0 msgr2=0x7f703015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70300cf0f0 0x7f703015c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 -- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7030071c70 msgr2=0x7f703006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7030071c70 0x7f703006ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 -- 172.21.15.146:0/1954092562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703006fe70 con 0x7f70300b2cd0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70355ec640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 0x7f70300cb660 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f702002a1e0 tx=0x7f70200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020041070 con 0x7f70300b2cd0 2024-04-25T15:21:23.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f702003c870 con 0x7f70300b2cd0 2024-04-25T15:21:23.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703006f8b0 con 0x7f70300b2cd0 2024-04-25T15:21:23.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020037040 con 0x7f70300b2cd0 2024-04-25T15:21:23.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.951+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703015f760 con 0x7f70300b2cd0 2024-04-25T15:21:23.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.952+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703006a0e0 con 0x7f70300b2cd0 2024-04-25T15:21:23.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.953+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7020044070 con 0x7f70300b2cd0 2024-04-25T15:21:23.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.953+0000 7f70267fc640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70140744b0 0x7f70140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:23.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.954+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f70200b5ed0 con 0x7f70300b2cd0 2024-04-25T15:21:23.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.954+0000 7f7035ded640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70140744b0 0x7f70140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:23.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.954+0000 7f7035ded640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70140744b0 0x7f70140769c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f70300ce350 tx=0x7f702c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:23.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:23.958+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f702007ff10 con 0x7f70300b2cd0 2024-04-25T15:21:24.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.136+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f70300ca270 con 0x7f70300b2cd0 2024-04-25T15:21:24.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.136+0000 7f70267fc640 1 -- 172.21.15.146:0/1954092562 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7f7020084210 con 0x7f70300b2cd0 2024-04-25T15:21:24.138 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:21:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70140744b0 msgr2=0x7f70140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70140744b0 0x7f70140769c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f70300ce350 tx=0x7f702c009060 comp rx=0 tx=0).stop 2024-04-25T15:21:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 msgr2=0x7f70300cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 0x7f70300cb660 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f702002a1e0 tx=0x7f70200046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 shutdown_connections 2024-04-25T15:21:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70140744b0 0x7f70140769c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70300cf0f0 0x7f703015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70300b2cd0 0x7f70300cb660 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 --2- 172.21.15.146:0/1954092562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7030071c70 0x7f703006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 >> 172.21.15.146:0/1954092562 conn(0x7f70300b52e0 msgr2=0x7f70300b83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 shutdown_connections 2024-04-25T15:21:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.138+0000 7f7038051640 1 -- 172.21.15.146:0/1954092562 wait complete. 2024-04-25T15:21:24.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:21:24.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.258+0000 7f518069a640 1 Processor -- start 2024-04-25T15:21:24.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.258+0000 7f518069a640 1 -- start start 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f518069a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f51780b2ba0 0x7f51781645c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f518069a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5178164b00 0x7f51780d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f518069a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780b8eb0 con 0x7f517806ee60 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f518069a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5178161840 con 0x7f5178164b00 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f518069a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f517808d730 con 0x7f51780b2ba0 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517ec37640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5178164b00 0x7f51780d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517e436640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f517806ee60 0x7f5178163eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46350/0 (socket says 172.21.15.146:46350) 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517e436640 1 -- 172.21.15.146:0/3869093953 learned_addr learned my addr 172.21.15.146:0/3869093953 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517dc35640 1 --2- 172.21.15.146:0/3869093953 >> v2:172.21.15.177:3300/0 conn(0x7f51780b2ba0 0x7f51781645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517ec37640 1 -- 172.21.15.146:0/3869093953 >> v2:172.21.15.177:3300/0 conn(0x7f51780b2ba0 msgr2=0x7f51781645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517ec37640 1 --2- 172.21.15.146:0/3869093953 >> v2:172.21.15.177:3300/0 conn(0x7f51780b2ba0 0x7f51781645c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517ec37640 1 -- 172.21.15.146:0/3869093953 >> v1:172.21.15.146:6789/0 conn(0x7f517806ee60 legacy=0x7f5178163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517d434640 1 -- 172.21.15.146:0/3869093953 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2089321782 0 0) 0x7f51780b8eb0 con 0x7f517806ee60 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517ec37640 1 -- 172.21.15.146:0/3869093953 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517808ce00 con 0x7f5178164b00 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.260+0000 7f517dc35640 1 --2- 172.21.15.146:0/3869093953 >> v2:172.21.15.177:3300/0 conn(0x7f51780b2ba0 0x7f51781645c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.261+0000 7f517ec37640 1 --2- 172.21.15.146:0/3869093953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5178164b00 0x7f51780d1b60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f51780b5290 tx=0x7f517002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab65287fcef9b05a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.261+0000 7f517d434640 1 -- 172.21.15.146:0/3869093953 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170037070 con 0x7f5178164b00 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.261+0000 7f517d434640 1 -- 172.21.15.146:0/3869093953 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5170032440 con 0x7f5178164b00 2024-04-25T15:21:24.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.261+0000 7f517d434640 1 -- 172.21.15.146:0/3869093953 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170032810 con 0x7f5178164b00 2024-04-25T15:21:24.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 -- 172.21.15.146:0/3869093953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5178164b00 msgr2=0x7f51780d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 --2- 172.21.15.146:0/3869093953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5178164b00 0x7f51780d1b60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f51780b5290 tx=0x7f517002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 -- 172.21.15.146:0/3869093953 shutdown_connections 2024-04-25T15:21:24.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 --2- 172.21.15.146:0/3869093953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5178164b00 0x7f51780d1b60 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 --2- 172.21.15.146:0/3869093953 >> v2:172.21.15.177:3300/0 conn(0x7f51780b2ba0 0x7f51781645c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 -- 172.21.15.146:0/3869093953 >> 172.21.15.146:0/3869093953 conn(0x7f51780b52e0 msgr2=0x7f51780b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:24.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 -- 172.21.15.146:0/3869093953 shutdown_connections 2024-04-25T15:21:24.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.262+0000 7f518069a640 1 -- 172.21.15.146:0/3869093953 wait complete. 2024-04-25T15:21:24.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.264+0000 7f518069a640 1 Processor -- start 2024-04-25T15:21:24.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.264+0000 7f518069a640 1 -- start start 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f518069a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f517806ee60 0x7f51780b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f518069a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 0x7f5178162f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f518069a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5178164b00 0x7f517815c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f518069a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f517808d3c0 con 0x7f51780b2ba0 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f518069a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f517806f440 con 0x7f5178164b00 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f518069a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f517808dbc0 con 0x7f517806ee60 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 0x7f5178162f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 0x7f5178162f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57028/0 (socket says 172.21.15.146:57028) 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 -- 172.21.15.146:0/3538540245 learned_addr learned my addr 172.21.15.146:0/3538540245 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517ec37640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5178164b00 0x7f517815c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517e436640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f517806ee60 0x7f51780b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 -- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f517806ee60 msgr2=0x7f51780b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f517806ee60 0x7f51780b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 -- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5178164b00 msgr2=0x7f517815c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5178164b00 0x7f517815c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517dc35640 1 -- 172.21.15.146:0/3538540245 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517815bf60 con 0x7f51780b2ba0 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.265+0000 7f517ec37640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5178164b00 0x7f517815c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:24.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.266+0000 7f517dc35640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 0x7f5178162f50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f516800c820 tx=0x7f516800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.266+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168015070 con 0x7f51780b2ba0 2024-04-25T15:21:24.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.266+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5168007a30 con 0x7f51780b2ba0 2024-04-25T15:21:24.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.266+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517815c240 con 0x7f51780b2ba0 2024-04-25T15:21:24.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.266+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168007de0 con 0x7f51780b2ba0 2024-04-25T15:21:24.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.266+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5178162750 con 0x7f51780b2ba0 2024-04-25T15:21:24.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.267+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f517806a0e0 con 0x7f51780b2ba0 2024-04-25T15:21:24.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.268+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5168018070 con 0x7f51780b2ba0 2024-04-25T15:21:24.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.268+0000 7f516effd640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f515c0744b0 0x7f515c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.268+0000 7f517e436640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f515c0744b0 0x7f515c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.268+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f51680952e0 con 0x7f51780b2ba0 2024-04-25T15:21:24.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.268+0000 7f517e436640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f515c0744b0 0x7f515c0769c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5174001c90 tx=0x7f5174009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.273+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f516805f8e0 con 0x7f51780b2ba0 2024-04-25T15:21:24.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.460+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f51780d35d0 con 0x7f51780b2ba0 2024-04-25T15:21:24.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.461+0000 7f516effd640 1 -- 172.21.15.146:0/3538540245 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7f516805f1e0 con 0x7f51780b2ba0 2024-04-25T15:21:24.463 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f515c0744b0 msgr2=0x7f515c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f515c0744b0 0x7f515c0769c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5174001c90 tx=0x7f5174009040 comp rx=0 tx=0).stop 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 msgr2=0x7f5178162f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 0x7f5178162f50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f516800c820 tx=0x7f516800cce0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 shutdown_connections 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f515c0744b0 0x7f515c0769c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5178164b00 0x7f517815c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51780b2ba0 0x7f5178162f50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 --2- 172.21.15.146:0/3538540245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f517806ee60 0x7f51780b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 >> 172.21.15.146:0/3538540245 conn(0x7f51780b52e0 msgr2=0x7f51780cf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 shutdown_connections 2024-04-25T15:21:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.463+0000 7f518069a640 1 -- 172.21.15.146:0/3538540245 wait complete. 2024-04-25T15:21:24.476 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:24.476 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:24.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:24.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.638+0000 7f804049f640 1 Processor -- start 2024-04-25T15:21:24.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.638+0000 7f804049f640 1 -- start start 2024-04-25T15:21:24.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f804049f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380cfbe0 0x7f80380cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f804049f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f80380d19d0 0x7f80380d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f804049f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80380b3920 0x7f80380b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f804049f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f80380b5020 con 0x7f80380b3920 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f804049f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f80380b4a60 con 0x7f80380d19d0 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f804049f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f80380b4d40 con 0x7f80380cfbe0 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f803ea3c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80380b3920 0x7f80380b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f803ea3c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80380b3920 0x7f80380b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57030/0 (socket says 172.21.15.146:57030) 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f803ea3c640 1 -- 172.21.15.146:0/3589945389 learned_addr learned my addr 172.21.15.146:0/3589945389 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f803da3a640 1 --2- 172.21.15.146:0/3589945389 >> v2:172.21.15.177:3300/0 conn(0x7f80380d19d0 0x7f80380d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.639+0000 7f803e23b640 1 --2- 172.21.15.146:0/3589945389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380cfbe0 0x7f80380cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803ea3c640 1 -- 172.21.15.146:0/3589945389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380cfbe0 msgr2=0x7f80380cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803ea3c640 1 --2- 172.21.15.146:0/3589945389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380cfbe0 0x7f80380cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803ea3c640 1 -- 172.21.15.146:0/3589945389 >> v2:172.21.15.177:3300/0 conn(0x7f80380d19d0 msgr2=0x7f80380d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803ea3c640 1 --2- 172.21.15.146:0/3589945389 >> v2:172.21.15.177:3300/0 conn(0x7f80380d19d0 0x7f80380d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803ea3c640 1 -- 172.21.15.146:0/3589945389 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80380cead0 con 0x7f80380b3920 2024-04-25T15:21:24.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803da3a640 1 --2- 172.21.15.146:0/3589945389 >> v2:172.21.15.177:3300/0 conn(0x7f80380d19d0 0x7f80380d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:24.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803ea3c640 1 --2- 172.21.15.146:0/3589945389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380b3920 0x7f80380b3d30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f80380b51f0 tx=0x7f803002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff99c0500fdea117 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.640+0000 7f803d239640 1 -- 172.21.15.146:0/3589945389 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030036070 con 0x7f80380b3920 2024-04-25T15:21:24.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f803d239640 1 -- 172.21.15.146:0/3589945389 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8030031600 con 0x7f80380b3920 2024-04-25T15:21:24.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f803d239640 1 -- 172.21.15.146:0/3589945389 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80300319d0 con 0x7f80380b3920 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 -- 172.21.15.146:0/3589945389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380b3920 msgr2=0x7f80380b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 --2- 172.21.15.146:0/3589945389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380b3920 0x7f80380b3d30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f80380b51f0 tx=0x7f803002a730 comp rx=0 tx=0).stop 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 -- 172.21.15.146:0/3589945389 shutdown_connections 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 --2- 172.21.15.146:0/3589945389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380b3920 0x7f80380b3d30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 --2- 172.21.15.146:0/3589945389 >> v2:172.21.15.177:3300/0 conn(0x7f80380d19d0 0x7f80380d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 --2- 172.21.15.146:0/3589945389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380cfbe0 0x7f80380cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 -- 172.21.15.146:0/3589945389 >> 172.21.15.146:0/3589945389 conn(0x7f80380b73b0 msgr2=0x7f80380b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 -- 172.21.15.146:0/3589945389 shutdown_connections 2024-04-25T15:21:24.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.641+0000 7f804049f640 1 -- 172.21.15.146:0/3589945389 wait complete. 2024-04-25T15:21:24.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.643+0000 7f804049f640 1 Processor -- start 2024-04-25T15:21:24.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.643+0000 7f804049f640 1 -- start start 2024-04-25T15:21:24.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f804049f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380b3920 0x7f80380cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f804049f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 0x7f80381e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f804049f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80380d19d0 0x7f803814a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f804049f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f80380cf7a0 con 0x7f80380cfbe0 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f804049f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f80380d2320 con 0x7f80380d19d0 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f804049f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f803814b4b0 con 0x7f80380b3920 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f803da3a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 0x7f80381e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.644+0000 7f803da3a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 0x7f80381e6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57038/0 (socket says 172.21.15.146:57038) 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 -- 172.21.15.146:0/109742708 learned_addr learned my addr 172.21.15.146:0/109742708 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803ea3c640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80380d19d0 0x7f803814a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803e23b640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380b3920 0x7f80380cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 -- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380b3920 msgr2=0x7f80380cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380b3920 0x7f80380cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 -- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80380d19d0 msgr2=0x7f803814a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80380d19d0 0x7f803814a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 -- 172.21.15.146:0/109742708 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80380cb410 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f803da3a640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 0x7f80381e6aa0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f802800d7d0 tx=0x7f802800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028004030 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80280041d0 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80280096a0 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80380cb6f0 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.645+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80380cbcb0 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.646+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8038160650 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.647+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8028009840 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.647+0000 7f802effd640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f801c0744b0 0x7f801c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.647+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8028095640 con 0x7f80380cfbe0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.648+0000 7f803e23b640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f801c0744b0 0x7f801c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.648+0000 7f803e23b640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f801c0744b0 0x7f801c0769c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f803400b970 tx=0x7f8034008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.652+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8038160650 con 0x7f80380cfbe0 2024-04-25T15:21:24.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.770+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f802805b860 con 0x7f80380cfbe0 2024-04-25T15:21:24.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.832+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8038160450 con 0x7f80380cfbe0 2024-04-25T15:21:24.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.834+0000 7f802effd640 1 -- 172.21.15.146:0/109742708 <== mon.0 v2:172.21.15.146:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16990 (secure 0 0 0) 0x7f8038160450 con 0x7f80380cfbe0 2024-04-25T15:21:24.835 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:24.836 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":39,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:09.275980+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f801c0744b0 msgr2=0x7f801c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f801c0744b0 0x7f801c0769c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f803400b970 tx=0x7f8034008040 comp rx=0 tx=0).stop 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 msgr2=0x7f80381e6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 0x7f80381e6aa0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f802800d7d0 tx=0x7f802800dc90 comp rx=0 tx=0).stop 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 shutdown_connections 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f801c0744b0 0x7f801c0769c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80380d19d0 0x7f803814a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80380cfbe0 0x7f80381e6aa0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 --2- 172.21.15.146:0/109742708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f80380b3920 0x7f80380cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 >> 172.21.15.146:0/109742708 conn(0x7f80380b73b0 msgr2=0x7f803806edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 shutdown_connections 2024-04-25T15:21:24.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.836+0000 7f804049f640 1 -- 172.21.15.146:0/109742708 wait complete. 2024-04-25T15:21:24.849 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:24.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.959+0000 7f2856d38640 1 Processor -- start 2024-04-25T15:21:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.959+0000 7f2856d38640 1 -- start start 2024-04-25T15:21:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f2856d38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28500d2d90 0x7f285009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f2856d38640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500b1960 con 0x7f28500b2400 2024-04-25T15:21:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f2856d38640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500b3ce0 con 0x7f28500ccd40 2024-04-25T15:21:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f2856d38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f28500b1c00 con 0x7f28500d2d90 2024-04-25T15:21:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f28552d5640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f28500b2400 0x7f28500b2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46366/0 (socket says 172.21.15.146:46366) 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f28552d5640 1 -- 172.21.15.146:0/831466848 learned_addr learned my addr 172.21.15.146:0/831466848 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284ffff640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28500d2d90 0x7f285009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354769553 0 0) 0x7f28500b1960 con 0x7f28500b2400 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2838003520 con 0x7f28500b2400 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 680678799 0 0) 0x7f2838003520 con 0x7f28500b2400 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28500d2d90 msgr2=0x7f285009bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28500d2d90 0x7f285009bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 >> v1:172.21.15.177:6789/0 conn(0x7f28500ccd40 legacy=0x7f28500d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28500b1ee0 con 0x7f28500b2400 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f2848001a10 con 0x7f28500b2400 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 >> v1:172.21.15.146:6789/0 conn(0x7f28500b2400 legacy=0x7f28500b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.961+0000 7f284f7fe640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28380039e0 0x7f2838003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284f7fe640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2838004370 0x7f28380048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284f7fe640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28380050c0 0x7f283800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f28500b1c00 con 0x7f2838004370 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f283800bf60 con 0x7f28380050c0 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f283800c160 con 0x7f28380039e0 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284ffff640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2838004370 0x7f28380048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f2854ad4640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28380039e0 0x7f2838003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f28552d5640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28380050c0 0x7f283800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284ffff640 1 -- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28380039e0 msgr2=0x7f2838003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284ffff640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28380039e0 0x7f2838003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284ffff640 1 -- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28380050c0 msgr2=0x7f283800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:24.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284ffff640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28380050c0 0x7f283800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f284ffff640 1 -- 172.21.15.146:0/831466848 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2838003520 con 0x7f2838004370 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.962+0000 7f28552d5640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28380050c0 0x7f283800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f284ffff640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2838004370 0x7f28380048c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2844009a90 tx=0x7f2838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8117ea3a8e05800 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844036070 con 0x7f2838004370 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2844031400 con 0x7f2838004370 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f284f7fe640 1 -- 172.21.15.146:0/831466848 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844009690 con 0x7f2838004370 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f2856d38640 1 -- 172.21.15.146:0/831466848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2838004370 msgr2=0x7f28380048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f2856d38640 1 --2- 172.21.15.146:0/831466848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2838004370 0x7f28380048c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2844009a90 tx=0x7f2838005830 comp rx=0 tx=0).stop 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f28552d5640 1 -- 172.21.15.146:0/831466848 reap_dead start 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f2856d38640 1 -- 172.21.15.146:0/831466848 shutdown_connections 2024-04-25T15:21:24.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f2856d38640 1 -- 172.21.15.146:0/831466848 >> 172.21.15.146:0/831466848 conn(0x7f28500b7400 msgr2=0x7f28500b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.963+0000 7f2856d38640 1 -- 172.21.15.146:0/831466848 shutdown_connections 2024-04-25T15:21:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.964+0000 7f2856d38640 1 -- 172.21.15.146:0/831466848 wait complete. 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.965+0000 7f2856d38640 1 Processor -- start 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.965+0000 7f2856d38640 1 -- start start 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2838004370 0x7f28500cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 0x7f28500b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285015efb0 0x7f28500b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f28500b3ac0 con 0x7f285015eba0 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f28500b3940 con 0x7f285015efb0 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f285009b440 con 0x7f2838004370 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 0x7f28500b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 0x7f28500b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57056/0 (socket says 172.21.15.146:57056) 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 -- 172.21.15.146:0/1716917717 learned_addr learned my addr 172.21.15.146:0/1716917717 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2854ad4640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2838004370 0x7f28500cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f28552d5640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285015efb0 0x7f28500b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 -- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2838004370 msgr2=0x7f28500cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2838004370 0x7f28500cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 -- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285015efb0 msgr2=0x7f28500b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285015efb0 0x7f28500b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 -- 172.21.15.146:0/1716917717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28500c9e00 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284ffff640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 0x7f28500b2810 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2844034040 tx=0x7f2844004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844040020 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2844003a40 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28500ca0e0 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844036040 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.967+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28500ca6a0 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.968+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28500c9b70 con 0x7f285015eba0 2024-04-25T15:21:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.969+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f2844043020 con 0x7f285015eba0 2024-04-25T15:21:24.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.969+0000 7f284d7fa640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f283c074540 0x7f283c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:24.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.970+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f28440b5650 con 0x7f285015eba0 2024-04-25T15:21:24.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.970+0000 7f2854ad4640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f283c074540 0x7f283c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:24.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.970+0000 7f2854ad4640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f283c074540 0x7f283c076a50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2840007fd0 tx=0x7f2840009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:24.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:24.974+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f284404a070 con 0x7f285015eba0 2024-04-25T15:21:25.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.152+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f285006a0e0 con 0x7f285015eba0 2024-04-25T15:21:25.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.153+0000 7f284d7fa640 1 -- 172.21.15.146:0/1716917717 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f284407fc60 con 0x7f285015eba0 2024-04-25T15:21:25.155 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f283c074540 msgr2=0x7f283c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f283c074540 0x7f283c076a50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2840007fd0 tx=0x7f2840009040 comp rx=0 tx=0).stop 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 msgr2=0x7f28500b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 0x7f28500b2810 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2844034040 tx=0x7f2844004730 comp rx=0 tx=0).stop 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 shutdown_connections 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f283c074540 0x7f283c076a50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285015efb0 0x7f28500b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285015eba0 0x7f28500b2810 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 --2- 172.21.15.146:0/1716917717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2838004370 0x7f28500cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 >> 172.21.15.146:0/1716917717 conn(0x7f28500b7400 msgr2=0x7f28500710c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 shutdown_connections 2024-04-25T15:21:25.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.155+0000 7f2856d38640 1 -- 172.21.15.146:0/1716917717 wait complete. 2024-04-25T15:21:25.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.277+0000 7f4d4b18f640 1 Processor -- start 2024-04-25T15:21:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.277+0000 7f4d4b18f640 1 -- start start 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d4b18f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d4415e070 0x7f4d440cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d4b18f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d440d2680 0x7f4d440d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d4b18f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d440d2fd0 con 0x7f4d4409bd80 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d4b18f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f4d440ce920 con 0x7f4d440d2680 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d4b18f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4d440cec00 con 0x7f4d4415e070 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d43fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4d4409bd80 0x7f4d440ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46368/0 (socket says 172.21.15.146:46368) 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.278+0000 7f4d43fff640 1 -- 172.21.15.146:0/3832500435 learned_addr learned my addr 172.21.15.146:0/3832500435 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d4972c640 1 --2- 172.21.15.146:0/3832500435 >> v2:172.21.15.177:3300/0 conn(0x7f4d440d2680 0x7f4d440d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d48f2b640 1 --2- 172.21.15.146:0/3832500435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d4415e070 0x7f4d440cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d437fe640 1 -- 172.21.15.146:0/3832500435 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157303817 0 0) 0x7f4d30001110 con 0x7f4d4409bd80 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d437fe640 1 -- 172.21.15.146:0/3832500435 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d34003520 con 0x7f4d4409bd80 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d437fe640 1 -- 172.21.15.146:0/3832500435 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2541009007 0 0) 0x7f4d34003520 con 0x7f4d4409bd80 2024-04-25T15:21:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3832500435 >> v2:172.21.15.177:3300/0 conn(0x7f4d440d2680 msgr2=0x7f4d440d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d48f2b640 1 --2- 172.21.15.146:0/3832500435 >> v2:172.21.15.177:3300/0 conn(0x7f4d440d2680 0x7f4d440d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3832500435 >> v1:172.21.15.146:6789/0 conn(0x7f4d4409bd80 legacy=0x7f4d440ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.279+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3832500435 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d440ceee0 con 0x7f4d4415e070 2024-04-25T15:21:25.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.280+0000 7f4d48f2b640 1 --2- 172.21.15.146:0/3832500435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d4415e070 0x7f4d440cf9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009790 tx=0x7f4d3802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e91316f227b174 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.280+0000 7f4d437fe640 1 -- 172.21.15.146:0/3832500435 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38037070 con 0x7f4d4415e070 2024-04-25T15:21:25.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.280+0000 7f4d437fe640 1 -- 172.21.15.146:0/3832500435 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3802ab90 con 0x7f4d4415e070 2024-04-25T15:21:25.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.280+0000 7f4d437fe640 1 -- 172.21.15.146:0/3832500435 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30001840 con 0x7f4d4415e070 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3832500435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d4415e070 msgr2=0x7f4d440cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3832500435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d4415e070 0x7f4d440cf9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009790 tx=0x7f4d3802a100 comp rx=0 tx=0).stop 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3832500435 shutdown_connections 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3832500435 >> v2:172.21.15.177:3300/0 conn(0x7f4d440d2680 0x7f4d440d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3832500435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d4415e070 0x7f4d440cf9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3832500435 >> 172.21.15.146:0/3832500435 conn(0x7f4d440b7400 msgr2=0x7f4d440b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3832500435 shutdown_connections 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.281+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3832500435 wait complete. 2024-04-25T15:21:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.283+0000 7f4d4b18f640 1 Processor -- start 2024-04-25T15:21:25.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.283+0000 7f4d4b18f640 1 -- start start 2024-04-25T15:21:25.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4b18f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 0x7f4d4409fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4b18f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d440d2680 0x7f4d440cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4b18f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d4415e070 0x7f4d440cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4b18f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4d440cff00 con 0x7f4d4409bd80 2024-04-25T15:21:25.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4b18f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4d440cf4a0 con 0x7f4d4415e070 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4b18f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4406cde0 con 0x7f4d440d2680 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 0x7f4d4409fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 0x7f4d4409fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57072/0 (socket says 172.21.15.146:57072) 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3627360707 learned_addr learned my addr 172.21.15.146:0/3627360707 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d43fff640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d440d2680 0x7f4d440cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d4972c640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d4415e070 0x7f4d440cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d440d2680 msgr2=0x7f4d440cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d440d2680 0x7f4d440cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d4415e070 msgr2=0x7f4d440cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d4415e070 0x7f4d440cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.284+0000 7f4d48f2b640 1 -- 172.21.15.146:0/3627360707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d440c86a0 con 0x7f4d4409bd80 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.285+0000 7f4d48f2b640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 0x7f4d4409fd10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4d380098b0 tx=0x7f4d380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.285+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38037040 con 0x7f4d4409bd80 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.285+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d38005b90 con 0x7f4d4409bd80 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.285+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d440c8980 con 0x7f4d4409bd80 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.285+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38006db0 con 0x7f4d4409bd80 2024-04-25T15:21:25.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.285+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d440c8f40 con 0x7f4d4409bd80 2024-04-25T15:21:25.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.286+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d4406a0e0 con 0x7f4d4409bd80 2024-04-25T15:21:25.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.287+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f4d38006030 con 0x7f4d4409bd80 2024-04-25T15:21:25.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.287+0000 7f4d417fa640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d28074570 0x7f4d28076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.287+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4d380b60d0 con 0x7f4d4409bd80 2024-04-25T15:21:25.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.287+0000 7f4d43fff640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d28074570 0x7f4d28076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.287+0000 7f4d43fff640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d28074570 0x7f4d28076a80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d441f8b40 tx=0x7f4d300012d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.292+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d38080120 con 0x7f4d4409bd80 2024-04-25T15:21:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.465+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d440b3fd0 con 0x7f4d4409bd80 2024-04-25T15:21:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.466+0000 7f4d417fa640 1 -- 172.21.15.146:0/3627360707 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f4d38084420 con 0x7f4d4409bd80 2024-04-25T15:21:25.468 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d28074570 msgr2=0x7f4d28076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d28074570 0x7f4d28076a80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d441f8b40 tx=0x7f4d300012d0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 msgr2=0x7f4d4409fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 0x7f4d4409fd10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4d380098b0 tx=0x7f4d380046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 shutdown_connections 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d28074570 0x7f4d28076a80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d4415e070 0x7f4d440cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d440d2680 0x7f4d440cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 --2- 172.21.15.146:0/3627360707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d4409bd80 0x7f4d4409fd10 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 >> 172.21.15.146:0/3627360707 conn(0x7f4d440b7400 msgr2=0x7f4d440b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 shutdown_connections 2024-04-25T15:21:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.468+0000 7f4d4b18f640 1 -- 172.21.15.146:0/3627360707 wait complete. 2024-04-25T15:21:25.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.590+0000 7f154cf5e640 1 Processor -- start 2024-04-25T15:21:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.590+0000 7f154cf5e640 1 -- start start 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154cf5e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f154815e180 0x7f15480cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154cf5e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154808d210 0x7f15480d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154cf5e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15480ce5a0 con 0x7f15480cfe20 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154cf5e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f15480ce840 con 0x7f154815e180 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154cf5e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f15480ceb20 con 0x7f154808d210 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154659c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f154815e180 0x7f15480cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f1545d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f15480cfe20 0x7f154808cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46376/0 (socket says 172.21.15.146:46376) 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f1545d9b640 1 -- 172.21.15.146:0/4174016155 learned_addr learned my addr 172.21.15.146:0/4174016155 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f1546d9d640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154808d210 0x7f15480d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154659c640 1 -- 172.21.15.146:0/4174016155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154808d210 msgr2=0x7f15480d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154659c640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154808d210 0x7f15480d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154659c640 1 -- 172.21.15.146:0/4174016155 >> v1:172.21.15.146:6789/0 conn(0x7f15480cfe20 legacy=0x7f154808cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154659c640 1 -- 172.21.15.146:0/4174016155 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15480cee00 con 0x7f154815e180 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f154559a640 1 -- 172.21.15.146:0/4174016155 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2851917280 0 0) 0x7f15480ce5a0 con 0x7f15480cfe20 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.592+0000 7f1546d9d640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154808d210 0x7f15480d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.593+0000 7f154659c640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f154815e180 0x7f15480cf8e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1538009de0 tx=0x7f153802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30db7664ed069d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.593+0000 7f154559a640 1 -- 172.21.15.146:0/4174016155 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538007740 con 0x7f154815e180 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.593+0000 7f154559a640 1 -- 172.21.15.146:0/4174016155 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15380078e0 con 0x7f154815e180 2024-04-25T15:21:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.593+0000 7f154559a640 1 -- 172.21.15.146:0/4174016155 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538007cb0 con 0x7f154815e180 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 -- 172.21.15.146:0/4174016155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f154815e180 msgr2=0x7f15480cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f154815e180 0x7f15480cf8e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1538009de0 tx=0x7f153802a6c0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 -- 172.21.15.146:0/4174016155 shutdown_connections 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154808d210 0x7f15480d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 --2- 172.21.15.146:0/4174016155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f154815e180 0x7f15480cf8e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 -- 172.21.15.146:0/4174016155 >> 172.21.15.146:0/4174016155 conn(0x7f15480b52e0 msgr2=0x7f15480b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 -- 172.21.15.146:0/4174016155 shutdown_connections 2024-04-25T15:21:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.594+0000 7f154cf5e640 1 -- 172.21.15.146:0/4174016155 wait complete. 2024-04-25T15:21:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.596+0000 7f154cf5e640 1 Processor -- start 2024-04-25T15:21:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.596+0000 7f154cf5e640 1 -- start start 2024-04-25T15:21:25.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154cf5e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f154808d210 0x7f15480b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154cf5e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 0x7f15480ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154cf5e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154815e180 0x7f15480728d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154cf5e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f154815e590 con 0x7f15480cfe20 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154cf5e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15480cf3c0 con 0x7f154808d210 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154cf5e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f15480d2910 con 0x7f154815e180 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f1545d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 0x7f15480ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f1545d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 0x7f15480ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57098/0 (socket says 172.21.15.146:57098) 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f1545d9b640 1 -- 172.21.15.146:0/2865581155 learned_addr learned my addr 172.21.15.146:0/2865581155 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:25.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.597+0000 7f154659c640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f154808d210 0x7f15480b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1546d9d640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154815e180 0x7f15480728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1545d9b640 1 -- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154815e180 msgr2=0x7f15480728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1545d9b640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154815e180 0x7f15480728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1545d9b640 1 -- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f154808d210 msgr2=0x7f15480b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1545d9b640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f154808d210 0x7f15480b8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1545d9b640 1 -- 172.21.15.146:0/2865581155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15480b9370 con 0x7f15480cfe20 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1545d9b640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 0x7f15480ca7c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c3f0 tx=0x7f153c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c0074e0 con 0x7f15480cfe20 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f153c0047e0 con 0x7f15480cfe20 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15480cb4f0 con 0x7f15480cfe20 2024-04-25T15:21:25.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c004c10 con 0x7f15480cfe20 2024-04-25T15:21:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.598+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15480cbab0 con 0x7f15480cfe20 2024-04-25T15:21:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.599+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f154806a0e0 con 0x7f15480cfe20 2024-04-25T15:21:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.600+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f153c011070 con 0x7f15480cfe20 2024-04-25T15:21:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.600+0000 7f1536ffd640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1528074570 0x7f1528076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.600+0000 7f154659c640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1528074570 0x7f1528076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.600+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f153c094e60 con 0x7f15480cfe20 2024-04-25T15:21:25.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.601+0000 7f154659c640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1528074570 0x7f1528076a80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1538034000 tx=0x7f153802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.605+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f153c05ee20 con 0x7f15480cfe20 2024-04-25T15:21:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.783+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f154808cff0 con 0x7f15480cfe20 2024-04-25T15:21:25.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.783+0000 7f1536ffd640 1 -- 172.21.15.146:0/2865581155 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7f153c063120 con 0x7f15480cfe20 2024-04-25T15:21:25.785 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1528074570 msgr2=0x7f1528076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1528074570 0x7f1528076a80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1538034000 tx=0x7f153802f000 comp rx=0 tx=0).stop 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 msgr2=0x7f15480ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 0x7f15480ca7c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c3f0 tx=0x7f153c00c8b0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 shutdown_connections 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1528074570 0x7f1528076a80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f154815e180 0x7f15480728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15480cfe20 0x7f15480ca7c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 --2- 172.21.15.146:0/2865581155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f154808d210 0x7f15480b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 >> 172.21.15.146:0/2865581155 conn(0x7f15480b52e0 msgr2=0x7f15480b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:25.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 shutdown_connections 2024-04-25T15:21:25.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.785+0000 7f154cf5e640 1 -- 172.21.15.146:0/2865581155 wait complete. 2024-04-25T15:21:25.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:21:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.905+0000 7faa58292640 1 Processor -- start 2024-04-25T15:21:25.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.905+0000 7faa58292640 1 -- start start 2024-04-25T15:21:25.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa58292640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa500d0730 0x7faa500d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa58292640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa5006ccc0 0x7faa5006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa58292640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa500b9200 con 0x7faa500d4f60 2024-04-25T15:21:25.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa58292640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7faa5006c550 con 0x7faa5006ccc0 2024-04-25T15:21:25.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa58292640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faa5006c830 con 0x7faa500d0730 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa5602e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7faa500d4f60 0x7faa500d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46390/0 (socket says 172.21.15.146:46390) 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.906+0000 7faa5602e640 1 -- 172.21.15.146:0/3771634647 learned_addr learned my addr 172.21.15.146:0/3771634647 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5682f640 1 --2- 172.21.15.146:0/3771634647 >> v2:172.21.15.146:3300/0 conn(0x7faa5006ccc0 0x7faa5006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5582d640 1 --2- 172.21.15.146:0/3771634647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa500d0730 0x7faa500d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5502c640 1 -- 172.21.15.146:0/3771634647 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110851897 0 0) 0x7faa4c001110 con 0x7faa500d4f60 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5502c640 1 -- 172.21.15.146:0/3771634647 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa3c003520 con 0x7faa500d4f60 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5682f640 1 -- 172.21.15.146:0/3771634647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa500d0730 msgr2=0x7faa500d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5682f640 1 --2- 172.21.15.146:0/3771634647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa500d0730 0x7faa500d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5682f640 1 -- 172.21.15.146:0/3771634647 >> v1:172.21.15.146:6789/0 conn(0x7faa500d4f60 legacy=0x7faa500d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.907+0000 7faa5682f640 1 -- 172.21.15.146:0/3771634647 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa500d1070 con 0x7faa5006ccc0 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa5682f640 1 --2- 172.21.15.146:0/3771634647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa5006ccc0 0x7faa5006d0d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faa48009d90 tx=0x7faa4802a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c09d9fd65678341 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa5502c640 1 -- 172.21.15.146:0/3771634647 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48007740 con 0x7faa5006ccc0 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa5502c640 1 -- 172.21.15.146:0/3771634647 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa480078e0 con 0x7faa5006ccc0 2024-04-25T15:21:25.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa5502c640 1 -- 172.21.15.146:0/3771634647 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48007cb0 con 0x7faa5006ccc0 2024-04-25T15:21:25.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa58292640 1 -- 172.21.15.146:0/3771634647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa5006ccc0 msgr2=0x7faa5006d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa58292640 1 --2- 172.21.15.146:0/3771634647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa5006ccc0 0x7faa5006d0d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faa48009d90 tx=0x7faa4802a530 comp rx=0 tx=0).stop 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa58292640 1 -- 172.21.15.146:0/3771634647 shutdown_connections 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa58292640 1 --2- 172.21.15.146:0/3771634647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa5006ccc0 0x7faa5006d0d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.908+0000 7faa58292640 1 --2- 172.21.15.146:0/3771634647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa500d0730 0x7faa500d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.909+0000 7faa58292640 1 -- 172.21.15.146:0/3771634647 >> 172.21.15.146:0/3771634647 conn(0x7faa500d2450 msgr2=0x7faa500d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.909+0000 7faa58292640 1 -- 172.21.15.146:0/3771634647 shutdown_connections 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.909+0000 7faa58292640 1 -- 172.21.15.146:0/3771634647 wait complete. 2024-04-25T15:21:25.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.910+0000 7faa58292640 1 Processor -- start 2024-04-25T15:21:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.910+0000 7faa58292640 1 -- start start 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5006ccc0 0x7faa5015c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 0x7faa500cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa501fa5e0 0x7faa501f9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faa500b5150 con 0x7faa500d4f60 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa500d15f0 con 0x7faa501fa5e0 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faa500b52d0 con 0x7faa5006ccc0 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 0x7faa500cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 0x7faa500cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57104/0 (socket says 172.21.15.146:57104) 2024-04-25T15:21:25.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 -- 172.21.15.146:0/723115450 learned_addr learned my addr 172.21.15.146:0/723115450 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 -- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5006ccc0 msgr2=0x7faa5015c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5602e640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5006ccc0 0x7faa5015c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5006ccc0 0x7faa5015c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 -- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa501fa5e0 msgr2=0x7faa501f9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa501fa5e0 0x7faa501f9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 -- 172.21.15.146:0/723115450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5015ced0 con 0x7faa500d4f60 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa5582d640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 0x7faa500cead0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e8a0 tx=0x7faa4000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40017070 con 0x7faa500d4f60 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4000c940 con 0x7faa500d4f60 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5015d1b0 con 0x7faa500d4f60 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4000cd30 con 0x7faa500d4f60 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.912+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa501e9ff0 con 0x7faa500d4f60 2024-04-25T15:21:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.913+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5015cc40 con 0x7faa500d4f60 2024-04-25T15:21:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.914+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7faa4001a070 con 0x7faa500d4f60 2024-04-25T15:21:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.914+0000 7faa46ffd640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa340745a0 0x7faa34076ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.914+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7faa40095570 con 0x7faa500d4f60 2024-04-25T15:21:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.914+0000 7faa5602e640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa340745a0 0x7faa34076ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.915+0000 7faa5602e640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa340745a0 0x7faa34076ab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0012b0 tx=0x7faa4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:25.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:25.919+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa4005fc10 con 0x7faa500d4f60 2024-04-25T15:21:26.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.106+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faa500d4100 con 0x7faa500d4f60 2024-04-25T15:21:26.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.106+0000 7faa46ffd640 1 -- 172.21.15.146:0/723115450 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7faa4005f510 con 0x7faa500d4f60 2024-04-25T15:21:26.108 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa340745a0 msgr2=0x7faa34076ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa340745a0 0x7faa34076ab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0012b0 tx=0x7faa4c009040 comp rx=0 tx=0).stop 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 msgr2=0x7faa500cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 0x7faa500cead0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e8a0 tx=0x7faa4000ed60 comp rx=0 tx=0).stop 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 shutdown_connections 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa340745a0 0x7faa34076ab0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa501fa5e0 0x7faa501f9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa500d4f60 0x7faa500cead0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 --2- 172.21.15.146:0/723115450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5006ccc0 0x7faa5015c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 >> 172.21.15.146:0/723115450 conn(0x7faa500d2450 msgr2=0x7faa500b93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 shutdown_connections 2024-04-25T15:21:26.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:26.109+0000 7faa58292640 1 -- 172.21.15.146:0/723115450 wait complete. 2024-04-25T15:21:26.121 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:21:26.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:26.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:26.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[51752]: fuse finished with error 0 and tester_r 0 2024-04-25T15:21:26.145 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:21:32.247 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:21:32.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:32.248 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:21:32.268 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:21:32.268 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:32.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.422+0000 7f408a540640 1 Processor -- start 2024-04-25T15:21:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.422+0000 7f408a540640 1 -- start start 2024-04-25T15:21:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f408a540640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f408a540640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f408406ccc0 0x7f408406d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f408a540640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40840b9200 con 0x7f40840d0020 2024-04-25T15:21:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f408a540640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f408406c550 con 0x7f408406ccc0 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f408a540640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f408406c830 con 0x7f40840d4f60 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f4088add640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f408406ccc0 0x7f408406d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f4088add640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f408406ccc0 0x7f408406d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40738/0 (socket says 172.21.15.146:40738) 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f4088add640 1 -- 172.21.15.146:0/258634220 learned_addr learned my addr 172.21.15.146:0/258634220 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.423+0000 7f40837fe640 1 --1- 172.21.15.146:0/258634220 >> v1:172.21.15.146:6789/0 conn(0x7f40840d0020 0x7f40840d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50162/0 (socket says 172.21.15.146:50162) 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4083fff640 1 --2- 172.21.15.146:0/258634220 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4088add640 1 -- 172.21.15.146:0/258634220 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 msgr2=0x7f40840d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4088add640 1 --2- 172.21.15.146:0/258634220 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4088add640 1 -- 172.21.15.146:0/258634220 >> v1:172.21.15.146:6789/0 conn(0x7f40840d0020 legacy=0x7f40840d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4082ffd640 1 -- 172.21.15.146:0/258634220 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343758363 0 0) 0x7f4078001110 con 0x7f40840d0020 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4088add640 1 -- 172.21.15.146:0/258634220 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40840d1070 con 0x7f408406ccc0 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.424+0000 7f4088add640 1 --2- 172.21.15.146:0/258634220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408406d0d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f407c009e50 tx=0x7f407c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946413679addf62c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f4082ffd640 1 -- 172.21.15.146:0/258634220 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c036070 con 0x7f408406ccc0 2024-04-25T15:21:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f4082ffd640 1 -- 172.21.15.146:0/258634220 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407c031600 con 0x7f408406ccc0 2024-04-25T15:21:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f4082ffd640 1 -- 172.21.15.146:0/258634220 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0319d0 con 0x7f408406ccc0 2024-04-25T15:21:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 -- 172.21.15.146:0/258634220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 msgr2=0x7f408406d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 --2- 172.21.15.146:0/258634220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408406d0d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f407c009e50 tx=0x7f407c02a610 comp rx=0 tx=0).stop 2024-04-25T15:21:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 -- 172.21.15.146:0/258634220 shutdown_connections 2024-04-25T15:21:32.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 --2- 172.21.15.146:0/258634220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408406d0d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 --2- 172.21.15.146:0/258634220 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 -- 172.21.15.146:0/258634220 >> 172.21.15.146:0/258634220 conn(0x7f40840d2450 msgr2=0x7f40840d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 -- 172.21.15.146:0/258634220 shutdown_connections 2024-04-25T15:21:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.425+0000 7f408a540640 1 -- 172.21.15.146:0/258634220 wait complete. 2024-04-25T15:21:32.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.427+0000 7f408a540640 1 Processor -- start 2024-04-25T15:21:32.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.427+0000 7f408a540640 1 -- start start 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.428+0000 7f408a540640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408415d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.428+0000 7f408a540640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40840d0020 0x7f408415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f408a540640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840bff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f408a540640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f40840b5150 con 0x7f408406ccc0 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f408a540640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40840d15f0 con 0x7f40840d0020 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f408a540640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f40840b52d0 con 0x7f40840d4f60 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408415d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408415d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40746/0 (socket says 172.21.15.146:40746) 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 -- 172.21.15.146:0/3388101884 learned_addr learned my addr 172.21.15.146:0/3388101884 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4088add640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f40837fe640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40840d0020 0x7f408415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 -- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 msgr2=0x7f40840bff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840bff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 -- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40840d0020 msgr2=0x7f408415deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40840d0020 0x7f408415deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 -- 172.21.15.146:0/3388101884 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408415e710 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4083fff640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408415d970 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f407400d5a0 tx=0x7f407400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074004240 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40740043e0 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074003830 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408415e9f0 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.429+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408415efb0 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.430+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408406a0e0 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.431+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f40740039d0 con 0x7f408406ccc0 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.432+0000 7f4080ff9640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4070074570 0x7f4070076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.432+0000 7f40837fe640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4070074570 0x7f4070076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.432+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f40740955b0 con 0x7f408406ccc0 2024-04-25T15:21:32.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.432+0000 7f40837fe640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4070074570 0x7f4070076a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4078000f60 tx=0x7f4078007d20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:32.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.436+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f407405fc30 con 0x7f408406ccc0 2024-04-25T15:21:32.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.611+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40840d4800 con 0x7f408406ccc0 2024-04-25T15:21:32.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.612+0000 7f4080ff9640 1 -- 172.21.15.146:0/3388101884 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16983 (secure 0 0 0) 0x7f407405f530 con 0x7f408406ccc0 2024-04-25T15:21:32.614 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:32.615 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":41,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:27.784978+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4070074570 msgr2=0x7f4070076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4070074570 0x7f4070076a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4078000f60 tx=0x7f4078007d20 comp rx=0 tx=0).stop 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 msgr2=0x7f408415d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408415d970 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f407400d5a0 tx=0x7f407400da60 comp rx=0 tx=0).stop 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 shutdown_connections 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4070074570 0x7f4070076a80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f40840d4f60 0x7f40840bff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40840d0020 0x7f408415deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 --2- 172.21.15.146:0/3388101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408406ccc0 0x7f408415d970 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 >> 172.21.15.146:0/3388101884 conn(0x7f40840d2450 msgr2=0x7f40840b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 shutdown_connections 2024-04-25T15:21:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.614+0000 7f408a540640 1 -- 172.21.15.146:0/3388101884 wait complete. 2024-04-25T15:21:32.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:32.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:32.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.737+0000 7f79bf805640 1 Processor -- start 2024-04-25T15:21:32.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.737+0000 7f79bf805640 1 -- start start 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bf805640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79b80ccd40 0x7f79b809bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bf805640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b80b1960 con 0x7f79b80b2400 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bf805640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b80b3ce0 con 0x7f79b80d2680 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bf805640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f79b80b1c00 con 0x7f79b80ccd40 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bdda2640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f79b80b2400 0x7f79b80b2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50172/0 (socket says 172.21.15.146:50172) 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bdda2640 1 -- 172.21.15.146:0/2765759459 learned_addr learned my addr 172.21.15.146:0/2765759459 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bd5a1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79b80ccd40 0x7f79b809bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bd5a1640 1 -- 172.21.15.146:0/2765759459 >> v1:172.21.15.177:6789/0 conn(0x7f79b80d2680 legacy=0x7f79b80d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bd5a1640 1 -- 172.21.15.146:0/2765759459 >> v1:172.21.15.146:6789/0 conn(0x7f79b80b2400 legacy=0x7f79b80b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79affff640 1 -- 172.21.15.146:0/2765759459 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 480569015 0 0) 0x7f79b80b1960 con 0x7f79b80b2400 2024-04-25T15:21:32.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.739+0000 7f79bd5a1640 1 -- 172.21.15.146:0/2765759459 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b80b1ee0 con 0x7f79b80ccd40 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.740+0000 7f79bd5a1640 1 --2- 172.21.15.146:0/2765759459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80ccd40 0x7f79b809bba0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009de0 tx=0x7f79b402a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f557bdf45194d3f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.740+0000 7f79affff640 1 -- 172.21.15.146:0/2765759459 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4007740 con 0x7f79b80ccd40 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.740+0000 7f79affff640 1 -- 172.21.15.146:0/2765759459 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b40078e0 con 0x7f79b80ccd40 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.740+0000 7f79affff640 1 -- 172.21.15.146:0/2765759459 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4007cb0 con 0x7f79b80ccd40 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 -- 172.21.15.146:0/2765759459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80ccd40 msgr2=0x7f79b809bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 --2- 172.21.15.146:0/2765759459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80ccd40 0x7f79b809bba0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009de0 tx=0x7f79b402a750 comp rx=0 tx=0).stop 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 -- 172.21.15.146:0/2765759459 shutdown_connections 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 --2- 172.21.15.146:0/2765759459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80ccd40 0x7f79b809bba0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 -- 172.21.15.146:0/2765759459 >> 172.21.15.146:0/2765759459 conn(0x7f79b80b7400 msgr2=0x7f79b80b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:32.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 -- 172.21.15.146:0/2765759459 shutdown_connections 2024-04-25T15:21:32.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.741+0000 7f79bf805640 1 -- 172.21.15.146:0/2765759459 wait complete. 2024-04-25T15:21:32.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.742+0000 7f79bf805640 1 Processor -- start 2024-04-25T15:21:32.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.743+0000 7f79bf805640 1 -- start start 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bf805640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80b2400 0x7f79b815fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bf805640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b80ccd40 0x7f79b8150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bf805640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 0x7f79b81f9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bf805640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f79b80b3ac0 con 0x7f79b80d2680 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bf805640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f79b80b3940 con 0x7f79b80ccd40 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bf805640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f79b809b440 con 0x7f79b80b2400 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 0x7f79b81f9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 0x7f79b81f9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40764/0 (socket says 172.21.15.146:40764) 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 -- 172.21.15.146:0/1236097114 learned_addr learned my addr 172.21.15.146:0/1236097114 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bd5a1640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80b2400 0x7f79b815fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bcda0640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b80ccd40 0x7f79b8150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 -- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80b2400 msgr2=0x7f79b815fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80b2400 0x7f79b815fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 -- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b80ccd40 msgr2=0x7f79b8150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b80ccd40 0x7f79b8150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 -- 172.21.15.146:0/1236097114 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b80cf150 con 0x7f79b80d2680 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.744+0000 7f79bdda2640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 0x7f79b81f9b40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f79b000c3e0 tx=0x7f79b000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.745+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0017070 con 0x7f79b80d2680 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.745+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b0005ce0 con 0x7f79b80d2680 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.745+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b8160560 con 0x7f79b80d2680 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.745+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b00072f0 con 0x7f79b80d2680 2024-04-25T15:21:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.745+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b81fa990 con 0x7f79b80d2680 2024-04-25T15:21:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.746+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b81fa400 con 0x7f79b80d2680 2024-04-25T15:21:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.746+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f79b001a070 con 0x7f79b80d2680 2024-04-25T15:21:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.747+0000 7f79adffb640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f799c074540 0x7f799c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.747+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f79b0094690 con 0x7f79b80d2680 2024-04-25T15:21:32.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.747+0000 7f79bd5a1640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f799c074540 0x7f799c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:32.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.747+0000 7f79bd5a1640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f799c074540 0x7f799c076a50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f79b4034000 tx=0x7f79b402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:32.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.751+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79b005e630 con 0x7f79b80d2680 2024-04-25T15:21:32.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.927+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79b80cfad0 con 0x7f79b80d2680 2024-04-25T15:21:32.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.927+0000 7f79adffb640 1 -- 172.21.15.146:0/1236097114 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f79b0062930 con 0x7f79b80d2680 2024-04-25T15:21:32.929 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:32.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f799c074540 msgr2=0x7f799c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f799c074540 0x7f799c076a50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f79b4034000 tx=0x7f79b402f000 comp rx=0 tx=0).stop 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 msgr2=0x7f79b81f9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 0x7f79b81f9b40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f79b000c3e0 tx=0x7f79b000c8a0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 shutdown_connections 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f799c074540 0x7f799c076a50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b80d2680 0x7f79b81f9b40 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b80ccd40 0x7f79b8150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 --2- 172.21.15.146:0/1236097114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80b2400 0x7f79b815fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.929+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 >> 172.21.15.146:0/1236097114 conn(0x7f79b80b7400 msgr2=0x7f79b80afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.930+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 shutdown_connections 2024-04-25T15:21:32.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:32.930+0000 7f79bf805640 1 -- 172.21.15.146:0/1236097114 wait complete. 2024-04-25T15:21:32.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:33.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.053+0000 7f81c348c640 1 Processor -- start 2024-04-25T15:21:33.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.053+0000 7f81c348c640 1 -- start start 2024-04-25T15:21:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c348c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f81bc160b00 0x7f81bc160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c348c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81bc15c440 0x7f81bc15c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c348c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81bc06f7b0 con 0x7f81bc06f2a0 2024-04-25T15:21:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c348c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f81bc0b1f30 con 0x7f81bc15c440 2024-04-25T15:21:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c348c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f81bc09b700 con 0x7f81bc160b00 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c1a29640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81bc15c440 0x7f81bc15c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c1a29640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81bc15c440 0x7f81bc15c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40770/0 (socket says 172.21.15.146:40770) 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c1228640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f81bc06f2a0 0x7f81bc0cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50174/0 (socket says 172.21.15.146:50174) 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.054+0000 7f81c1a29640 1 -- 172.21.15.146:0/3751177277 learned_addr learned my addr 172.21.15.146:0/3751177277 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c0a27640 1 --2- 172.21.15.146:0/3751177277 >> v2:172.21.15.177:3300/0 conn(0x7f81bc160b00 0x7f81bc160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c1a29640 1 -- 172.21.15.146:0/3751177277 >> v2:172.21.15.177:3300/0 conn(0x7f81bc160b00 msgr2=0x7f81bc160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c1a29640 1 --2- 172.21.15.146:0/3751177277 >> v2:172.21.15.177:3300/0 conn(0x7f81bc160b00 0x7f81bc160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c1a29640 1 -- 172.21.15.146:0/3751177277 >> v1:172.21.15.146:6789/0 conn(0x7f81bc06f2a0 legacy=0x7f81bc0cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81b3fff640 1 -- 172.21.15.146:0/3751177277 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437754756 0 0) 0x7f81bc06f7b0 con 0x7f81bc06f2a0 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c1a29640 1 -- 172.21.15.146:0/3751177277 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81bc09b9e0 con 0x7f81bc15c440 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c0a27640 1 --2- 172.21.15.146:0/3751177277 >> v2:172.21.15.177:3300/0 conn(0x7f81bc160b00 0x7f81bc160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81c1a29640 1 --2- 172.21.15.146:0/3751177277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc15c850 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f81bc06feb0 tx=0x7f81b402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f875bb5310e76ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81b3fff640 1 -- 172.21.15.146:0/3751177277 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b4037070 con 0x7f81bc15c440 2024-04-25T15:21:33.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81b3fff640 1 -- 172.21.15.146:0/3751177277 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81b4032440 con 0x7f81bc15c440 2024-04-25T15:21:33.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.055+0000 7f81b3fff640 1 -- 172.21.15.146:0/3751177277 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b4032810 con 0x7f81bc15c440 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 -- 172.21.15.146:0/3751177277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 msgr2=0x7f81bc15c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 --2- 172.21.15.146:0/3751177277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc15c850 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f81bc06feb0 tx=0x7f81b402a270 comp rx=0 tx=0).stop 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 -- 172.21.15.146:0/3751177277 shutdown_connections 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 --2- 172.21.15.146:0/3751177277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc15c850 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 --2- 172.21.15.146:0/3751177277 >> v2:172.21.15.177:3300/0 conn(0x7f81bc160b00 0x7f81bc160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 -- 172.21.15.146:0/3751177277 >> 172.21.15.146:0/3751177277 conn(0x7f81bc0b7490 msgr2=0x7f81bc0b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 -- 172.21.15.146:0/3751177277 shutdown_connections 2024-04-25T15:21:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.056+0000 7f81c348c640 1 -- 172.21.15.146:0/3751177277 wait complete. 2024-04-25T15:21:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.058+0000 7f81c348c640 1 Processor -- start 2024-04-25T15:21:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.058+0000 7f81c348c640 1 -- start start 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c348c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81bc06f2a0 0x7f81bc154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c348c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c348c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81bc160b00 0x7f81bc0d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c348c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc15cf10 con 0x7f81bc15c440 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c348c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81bc15cd90 con 0x7f81bc06f2a0 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c348c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f81bc15d090 con 0x7f81bc160b00 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c0a27640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c0a27640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40776/0 (socket says 172.21.15.146:40776) 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.059+0000 7f81c0a27640 1 -- 172.21.15.146:0/3621975687 learned_addr learned my addr 172.21.15.146:0/3621975687 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c1228640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81bc06f2a0 0x7f81bc154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c1a29640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81bc160b00 0x7f81bc0d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c0a27640 1 -- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81bc160b00 msgr2=0x7f81bc0d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c0a27640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81bc160b00 0x7f81bc0d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c0a27640 1 -- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81bc06f2a0 msgr2=0x7f81bc154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c0a27640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81bc06f2a0 0x7f81bc154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c0a27640 1 -- 172.21.15.146:0/3621975687 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81bc15a8e0 con 0x7f81bc15c440 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c0a27640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc155c00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00c820 tx=0x7f81ac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac015070 con 0x7f81bc15c440 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ac007a30 con 0x7f81bc15c440 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81bc15abc0 con 0x7f81bc15c440 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac007de0 con 0x7f81bc15c440 2024-04-25T15:21:33.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.060+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81bc15b180 con 0x7f81bc15c440 2024-04-25T15:21:33.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.061+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81bc06a0e0 con 0x7f81bc15c440 2024-04-25T15:21:33.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.062+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f81ac018070 con 0x7f81bc15c440 2024-04-25T15:21:33.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.062+0000 7f81b1ffb640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81a0074540 0x7f81a0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.062+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f81ac0954a0 con 0x7f81bc15c440 2024-04-25T15:21:33.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.062+0000 7f81c1228640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81a0074540 0x7f81a0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.063+0000 7f81c1228640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81a0074540 0x7f81a0076a50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f81b8007ed0 tx=0x7f81b8007e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.067+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81ac05f980 con 0x7f81bc15c440 2024-04-25T15:21:33.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.242+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81bc0b26c0 con 0x7f81bc15c440 2024-04-25T15:21:33.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.242+0000 7f81b1ffb640 1 -- 172.21.15.146:0/3621975687 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7f81ac05f280 con 0x7f81bc15c440 2024-04-25T15:21:33.244 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:33.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81a0074540 msgr2=0x7f81a0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81a0074540 0x7f81a0076a50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f81b8007ed0 tx=0x7f81b8007e60 comp rx=0 tx=0).stop 2024-04-25T15:21:33.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 msgr2=0x7f81bc155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc155c00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00c820 tx=0x7f81ac00cce0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 shutdown_connections 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81a0074540 0x7f81a0076a50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81bc160b00 0x7f81bc0d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81bc15c440 0x7f81bc155c00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 --2- 172.21.15.146:0/3621975687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81bc06f2a0 0x7f81bc154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 >> 172.21.15.146:0/3621975687 conn(0x7f81bc0b7490 msgr2=0x7f81bc0cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 shutdown_connections 2024-04-25T15:21:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.244+0000 7f81c348c640 1 -- 172.21.15.146:0/3621975687 wait complete. 2024-04-25T15:21:33.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.366+0000 7f17a2d3f640 1 Processor -- start 2024-04-25T15:21:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.366+0000 7f17a2d3f640 1 -- start start 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a2d3f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f179c160210 0x7f179c09b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a2d3f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f179c06dce0 0x7f179c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a2d3f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f179c06c550 con 0x7f179c06d8d0 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a2d3f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f179c06d440 con 0x7f179c06dce0 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a2d3f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f179c06c830 con 0x7f179c160210 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f179c06dce0 0x7f179c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f179c06dce0 0x7f179c0af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40786/0 (socket says 172.21.15.146:40786) 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 -- 172.21.15.146:0/1410398274 learned_addr learned my addr 172.21.15.146:0/1410398274 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a0adb640 1 --2- 172.21.15.146:0/1410398274 >> v2:172.21.15.177:3300/0 conn(0x7f179c160210 0x7f179c09b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 -- 172.21.15.146:0/1410398274 >> v2:172.21.15.177:3300/0 conn(0x7f179c160210 msgr2=0x7f179c09b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:33.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 --2- 172.21.15.146:0/1410398274 >> v2:172.21.15.177:3300/0 conn(0x7f179c160210 0x7f179c09b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 -- 172.21.15.146:0/1410398274 >> v1:172.21.15.177:6789/0 conn(0x7f179c06d8d0 legacy=0x7f179c0af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.368+0000 7f17a12dc640 1 -- 172.21.15.146:0/1410398274 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c06cb10 con 0x7f179c06dce0 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.369+0000 7f17a12dc640 1 --2- 172.21.15.146:0/1410398274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06dce0 0x7f179c0af840 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f179c06d5c0 tx=0x7f179402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b475a3d45cd9df39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.369+0000 7f179b7fe640 1 -- 172.21.15.146:0/1410398274 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794036070 con 0x7f179c06dce0 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.369+0000 7f179b7fe640 1 -- 172.21.15.146:0/1410398274 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1794031600 con 0x7f179c06dce0 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.369+0000 7f179b7fe640 1 -- 172.21.15.146:0/1410398274 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17940319d0 con 0x7f179c06dce0 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 -- 172.21.15.146:0/1410398274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06dce0 msgr2=0x7f179c0af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/1410398274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06dce0 0x7f179c0af840 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f179c06d5c0 tx=0x7f179402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 -- 172.21.15.146:0/1410398274 shutdown_connections 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/1410398274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06dce0 0x7f179c0af840 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/1410398274 >> v2:172.21.15.177:3300/0 conn(0x7f179c160210 0x7f179c09b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 -- 172.21.15.146:0/1410398274 >> 172.21.15.146:0/1410398274 conn(0x7f179c073a30 msgr2=0x7f179c0b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:33.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 -- 172.21.15.146:0/1410398274 shutdown_connections 2024-04-25T15:21:33.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.370+0000 7f17a2d3f640 1 -- 172.21.15.146:0/1410398274 wait complete. 2024-04-25T15:21:33.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.371+0000 7f17a2d3f640 1 Processor -- start 2024-04-25T15:21:33.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.371+0000 7f17a2d3f640 1 -- start start 2024-04-25T15:21:33.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a2d3f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 0x7f179c150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a2d3f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f179c06dce0 0x7f179c1536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a2d3f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c160210 0x7f179c153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a2d3f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f179c163cd0 con 0x7f179c06d8d0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a2d3f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f179c06d0d0 con 0x7f179c160210 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a2d3f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f179c163e50 con 0x7f179c06dce0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 0x7f179c150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 0x7f179c150640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40794/0 (socket says 172.21.15.146:40794) 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 -- 172.21.15.146:0/4208749062 learned_addr learned my addr 172.21.15.146:0/4208749062 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a12dc640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c160210 0x7f179c153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f179bfff640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f179c06dce0 0x7f179c1536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 -- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f179c06dce0 msgr2=0x7f179c1536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f179c06dce0 0x7f179c1536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 -- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c160210 msgr2=0x7f179c153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c160210 0x7f179c153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f17a0adb640 1 -- 172.21.15.146:0/4208749062 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c154490 con 0x7f179c06d8d0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.373+0000 7f179bfff640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f179c06dce0 0x7f179c1536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.374+0000 7f17a0adb640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 0x7f179c150640 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f178c00e870 tx=0x7f178c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.374+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c017070 con 0x7f179c06d8d0 2024-04-25T15:21:33.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.374+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178c00c940 con 0x7f179c06d8d0 2024-04-25T15:21:33.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.374+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c155ba0 con 0x7f179c06d8d0 2024-04-25T15:21:33.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.374+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c00cd10 con 0x7f179c06d8d0 2024-04-25T15:21:33.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.374+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c157700 con 0x7f179c06d8d0 2024-04-25T15:21:33.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.375+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f179c157960 con 0x7f179c06d8d0 2024-04-25T15:21:33.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.375+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f178c01a070 con 0x7f179c06d8d0 2024-04-25T15:21:33.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.376+0000 7f17997fa640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1788074540 0x7f1788076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.376+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f178c095540 con 0x7f179c06d8d0 2024-04-25T15:21:33.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.376+0000 7f179bfff640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1788074540 0x7f1788076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.376+0000 7f179bfff640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1788074540 0x7f1788076a50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f179c150f70 tx=0x7f1790009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.380+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f178c05f4f0 con 0x7f179c06d8d0 2024-04-25T15:21:33.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.556+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f179c06a0e0 con 0x7f179c06d8d0 2024-04-25T15:21:33.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.556+0000 7f17997fa640 1 -- 172.21.15.146:0/4208749062 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7f178c0637f0 con 0x7f179c06d8d0 2024-04-25T15:21:33.558 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:21:33.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1788074540 msgr2=0x7f1788076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1788074540 0x7f1788076a50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f179c150f70 tx=0x7f1790009040 comp rx=0 tx=0).stop 2024-04-25T15:21:33.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 msgr2=0x7f179c150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 0x7f179c150640 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f178c00e870 tx=0x7f178c00ed30 comp rx=0 tx=0).stop 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 shutdown_connections 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1788074540 0x7f1788076a50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c160210 0x7f179c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f179c06dce0 0x7f179c1536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 --2- 172.21.15.146:0/4208749062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c06d8d0 0x7f179c150640 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.558+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 >> 172.21.15.146:0/4208749062 conn(0x7f179c073a30 msgr2=0x7f179c15a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.559+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 shutdown_connections 2024-04-25T15:21:33.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.559+0000 7f17a2d3f640 1 -- 172.21.15.146:0/4208749062 wait complete. 2024-04-25T15:21:33.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:21:33.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.677+0000 7f091904c640 1 Processor -- start 2024-04-25T15:21:33.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.677+0000 7f091904c640 1 -- start start 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091904c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091904c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0914161b30 0x7f0914161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091904c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09140d1420 con 0x7f09140cf1d0 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091904c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f09140ced40 con 0x7f0914161b30 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091904c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f09140d1700 con 0x7f09140d1cc0 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091259c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091359e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0914161b30 0x7f0914161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f0912d9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f09140cf1d0 0x7f09140b7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:53440/0 (socket says 172.21.15.146:53440) 2024-04-25T15:21:33.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f0912d9d640 1 -- 172.21.15.146:0/1007083687 learned_addr learned my addr 172.21.15.146:0/1007083687 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:33.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091259c640 1 -- 172.21.15.146:0/1007083687 >> v2:172.21.15.177:3300/0 conn(0x7f0914161b30 msgr2=0x7f0914161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:33.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091259c640 1 --2- 172.21.15.146:0/1007083687 >> v2:172.21.15.177:3300/0 conn(0x7f0914161b30 0x7f0914161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091259c640 1 -- 172.21.15.146:0/1007083687 >> v1:172.21.15.177:6789/0 conn(0x7f09140cf1d0 legacy=0x7f09140b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091259c640 1 -- 172.21.15.146:0/1007083687 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09140d19e0 con 0x7f09140d1cc0 2024-04-25T15:21:33.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.679+0000 7f091359e640 1 --2- 172.21.15.146:0/1007083687 >> v2:172.21.15.177:3300/0 conn(0x7f0914161b30 0x7f0914161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:33.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.680+0000 7f091259c640 1 --2- 172.21.15.146:0/1007083687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140d20d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09080097a0 tx=0x7f090802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bd5d4afb7a64b8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.680+0000 7f0911d9b640 1 -- 172.21.15.146:0/1007083687 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908037070 con 0x7f09140d1cc0 2024-04-25T15:21:33.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.680+0000 7f0911d9b640 1 -- 172.21.15.146:0/1007083687 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0908032440 con 0x7f09140d1cc0 2024-04-25T15:21:33.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.680+0000 7f0911d9b640 1 -- 172.21.15.146:0/1007083687 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0908032810 con 0x7f09140d1cc0 2024-04-25T15:21:33.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 -- 172.21.15.146:0/1007083687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 msgr2=0x7f09140d20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 --2- 172.21.15.146:0/1007083687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140d20d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09080097a0 tx=0x7f090802a180 comp rx=0 tx=0).stop 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 -- 172.21.15.146:0/1007083687 shutdown_connections 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 --2- 172.21.15.146:0/1007083687 >> v2:172.21.15.177:3300/0 conn(0x7f0914161b30 0x7f0914161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 --2- 172.21.15.146:0/1007083687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140d20d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 -- 172.21.15.146:0/1007083687 >> 172.21.15.146:0/1007083687 conn(0x7f091409b2b0 msgr2=0x7f09140b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 -- 172.21.15.146:0/1007083687 shutdown_connections 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.681+0000 7f091904c640 1 -- 172.21.15.146:0/1007083687 wait complete. 2024-04-25T15:21:33.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.683+0000 7f091904c640 1 Processor -- start 2024-04-25T15:21:33.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.683+0000 7f091904c640 1 -- start start 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091904c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09140cf1d0 0x7f0914152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091904c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091904c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 0x7f0914160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091904c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f09140d2610 con 0x7f0914161b30 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091904c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09140cf5e0 con 0x7f09140cf1d0 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091904c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f091415e310 con 0x7f09140d1cc0 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.684+0000 7f091359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 0x7f0914160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 0x7f0914160770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40796/0 (socket says 172.21.15.146:40796) 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 -- 172.21.15.146:0/2131482958 learned_addr learned my addr 172.21.15.146:0/2131482958 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f0912d9d640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09140cf1d0 0x7f0914152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091259c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 -- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 msgr2=0x7f09140c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 -- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09140cf1d0 msgr2=0x7f0914152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09140cf1d0 0x7f0914152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 -- 172.21.15.146:0/2131482958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09140c9540 con 0x7f0914161b30 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091359e640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 0x7f0914160770 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f090c00efc0 tx=0x7f090c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c018020 con 0x7f0914161b30 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f090c014480 con 0x7f0914161b30 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09140c9820 con 0x7f0914161b30 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c014870 con 0x7f0914161b30 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.685+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09140c9de0 con 0x7f0914161b30 2024-04-25T15:21:33.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.686+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f091406a0e0 con 0x7f0914161b30 2024-04-25T15:21:33.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.687+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f090c005d10 con 0x7f0914161b30 2024-04-25T15:21:33.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.687+0000 7f08fb7fe640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0900074570 0x7f0900076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:33.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.687+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f090c0960d0 con 0x7f0914161b30 2024-04-25T15:21:33.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.687+0000 7f0912d9d640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0900074570 0x7f0900076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:33.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.688+0000 7f0912d9d640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0900074570 0x7f0900076a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0904000f60 tx=0x7f0904009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:33.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.692+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f090c022030 con 0x7f0914161b30 2024-04-25T15:21:33.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.883+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f09140d2a70 con 0x7f0914161b30 2024-04-25T15:21:33.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.883+0000 7f08fb7fe640 1 -- 172.21.15.146:0/2131482958 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7f090c010040 con 0x7f0914161b30 2024-04-25T15:21:33.885 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:21:33.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0900074570 msgr2=0x7f0900076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0900074570 0x7f0900076a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0904000f60 tx=0x7f0904009040 comp rx=0 tx=0).stop 2024-04-25T15:21:33.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 msgr2=0x7f0914160770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:33.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 0x7f0914160770 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f090c00efc0 tx=0x7f090c00c490 comp rx=0 tx=0).stop 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 shutdown_connections 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0900074570 0x7f0900076a80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0914161b30 0x7f0914160770 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09140d1cc0 0x7f09140c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 --2- 172.21.15.146:0/2131482958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09140cf1d0 0x7f0914152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 >> 172.21.15.146:0/2131482958 conn(0x7f091409b2b0 msgr2=0x7f09140b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 shutdown_connections 2024-04-25T15:21:33.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:33.885+0000 7f091904c640 1 -- 172.21.15.146:0/2131482958 wait complete. 2024-04-25T15:21:33.899 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:21:33.899 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:33.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:34.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.025+0000 7fa6ede49640 1 Processor -- start 2024-04-25T15:21:34.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.025+0000 7fa6ede49640 1 -- start start 2024-04-25T15:21:34.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6ede49640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa6e80cde20 0x7fa6e816a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6ede49640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6e80b4a60 con 0x7fa6e80d19d0 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6ede49640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6e80b4d40 con 0x7fa6e80cfbe0 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6ede49640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa6e80b5020 con 0x7fa6e80cde20 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e6ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa6e80d19d0 0x7fa6e80d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50190/0 (socket says 172.21.15.146:50190) 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e6ffd640 1 -- 172.21.15.146:0/3198209634 learned_addr learned my addr 172.21.15.146:0/3198209634 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e7fff640 1 --2- 172.21.15.146:0/3198209634 >> v2:172.21.15.146:3300/0 conn(0x7fa6e80cde20 0x7fa6e816a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e7fff640 1 -- 172.21.15.146:0/3198209634 >> v1:172.21.15.177:6789/0 conn(0x7fa6e80cfbe0 legacy=0x7fa6e80cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e7fff640 1 -- 172.21.15.146:0/3198209634 >> v1:172.21.15.146:6789/0 conn(0x7fa6e80d19d0 legacy=0x7fa6e80d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e7fff640 1 -- 172.21.15.146:0/3198209634 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e80cead0 con 0x7fa6e80cde20 2024-04-25T15:21:34.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.027+0000 7fa6e67fc640 1 -- 172.21.15.146:0/3198209634 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 40185465 0 0) 0x7fa6e80b4a60 con 0x7fa6e80d19d0 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.028+0000 7fa6e7fff640 1 --2- 172.21.15.146:0/3198209634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e816a970 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009d90 tx=0x7fa6e002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8845fed1383434 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.028+0000 7fa6e67fc640 1 -- 172.21.15.146:0/3198209634 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0007740 con 0x7fa6e80cde20 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.028+0000 7fa6e67fc640 1 -- 172.21.15.146:0/3198209634 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e00078e0 con 0x7fa6e80cde20 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.028+0000 7fa6e67fc640 1 -- 172.21.15.146:0/3198209634 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0007cb0 con 0x7fa6e80cde20 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 -- 172.21.15.146:0/3198209634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 msgr2=0x7fa6e816a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3198209634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e816a970 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009d90 tx=0x7fa6e002a6f0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 -- 172.21.15.146:0/3198209634 shutdown_connections 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3198209634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e816a970 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 -- 172.21.15.146:0/3198209634 >> 172.21.15.146:0/3198209634 conn(0x7fa6e80b73b0 msgr2=0x7fa6e80b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 -- 172.21.15.146:0/3198209634 shutdown_connections 2024-04-25T15:21:34.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.029+0000 7fa6ede49640 1 -- 172.21.15.146:0/3198209634 wait complete. 2024-04-25T15:21:34.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.030+0000 7fa6ede49640 1 Processor -- start 2024-04-25T15:21:34.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.031+0000 7fa6ede49640 1 -- start start 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6ede49640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e80c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6ede49640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e80cfbe0 0x7fa6e815e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6ede49640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e80d19d0 0x7fa6e814b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6ede49640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e816b030 con 0x7fa6e80cde20 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6ede49640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e816aeb0 con 0x7fa6e80d19d0 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6ede49640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa6e816b1b0 con 0x7fa6e80cfbe0 2024-04-25T15:21:34.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e80c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e80c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40808/0 (socket says 172.21.15.146:40808) 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 -- 172.21.15.146:0/3080820093 learned_addr learned my addr 172.21.15.146:0/3080820093 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 -- 172.21.15.146:0/3080820093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e80cfbe0 msgr2=0x7fa6e815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e80cfbe0 0x7fa6e815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 -- 172.21.15.146:0/3080820093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e80d19d0 msgr2=0x7fa6e814b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e80d19d0 0x7fa6e814b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 -- 172.21.15.146:0/3080820093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e815c780 con 0x7fa6e80cde20 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.032+0000 7fa6e77fe640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e80c8560 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c3a0 tx=0x7fa6d800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.033+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8017070 con 0x7fa6e80cde20 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.033+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d80052f0 con 0x7fa6e80cde20 2024-04-25T15:21:34.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.033+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e815ca60 con 0x7fa6e80cde20 2024-04-25T15:21:34.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.033+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8003640 con 0x7fa6e80cde20 2024-04-25T15:21:34.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.033+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e815d020 con 0x7fa6e80cde20 2024-04-25T15:21:34.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.034+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e806a0e0 con 0x7fa6e80cde20 2024-04-25T15:21:34.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.034+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa6d801a070 con 0x7fa6e80cde20 2024-04-25T15:21:34.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.035+0000 7fa6cffff640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa6d4074540 0x7fa6d4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.035+0000 7fa6e6ffd640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa6d4074540 0x7fa6d4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.035+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa6d8094750 con 0x7fa6e80cde20 2024-04-25T15:21:34.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.035+0000 7fa6e6ffd640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa6d4074540 0x7fa6d4076a50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc0079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.040+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6d805edd0 con 0x7fa6e80cde20 2024-04-25T15:21:34.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.216+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6e815de50 con 0x7fa6e80cde20 2024-04-25T15:21:34.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.217+0000 7fa6cffff640 1 -- 172.21.15.146:0/3080820093 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16983 (secure 0 0 0) 0x7fa6d805e6d0 con 0x7fa6e80cde20 2024-04-25T15:21:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:34.219 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":41,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:27.784978+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa6d4074540 msgr2=0x7fa6d4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa6d4074540 0x7fa6d4076a50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc0079c0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 msgr2=0x7fa6e80c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e80c8560 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c3a0 tx=0x7fa6d800c860 comp rx=0 tx=0).stop 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 shutdown_connections 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa6d4074540 0x7fa6d4076a50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e80d19d0 0x7fa6e814b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e80cfbe0 0x7fa6e815e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 --2- 172.21.15.146:0/3080820093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6e80cde20 0x7fa6e80c8560 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 >> 172.21.15.146:0/3080820093 conn(0x7fa6e80b73b0 msgr2=0x7fa6e80b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 shutdown_connections 2024-04-25T15:21:34.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.219+0000 7fa6ede49640 1 -- 172.21.15.146:0/3080820093 wait complete. 2024-04-25T15:21:34.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:34.233 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:34.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.342+0000 7f5bbdbbe640 1 Processor -- start 2024-04-25T15:21:34.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.342+0000 7f5bbdbbe640 1 -- start start 2024-04-25T15:21:34.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bbdbbe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bb80af840 0x7f5bb816adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bbdbbe640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb806c550 con 0x7f5bb8160210 2024-04-25T15:21:34.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bbdbbe640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb806c830 con 0x7f5bb809b2b0 2024-04-25T15:21:34.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bbdbbe640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb806dfe0 con 0x7f5bb80af840 2024-04-25T15:21:34.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb7fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bb80af840 0x7f5bb816adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb7fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bb80af840 0x7f5bb816adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40816/0 (socket says 172.21.15.146:40816) 2024-04-25T15:21:34.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb7fff640 1 -- 172.21.15.146:0/980211114 learned_addr learned my addr 172.21.15.146:0/980211114 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:34.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb77fe640 1 --1- 172.21.15.146:0/980211114 >> v1:172.21.15.146:6789/0 conn(0x7f5bb8160210 0x7f5bb806d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50192/0 (socket says 172.21.15.146:50192) 2024-04-25T15:21:34.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb7fff640 1 -- 172.21.15.146:0/980211114 >> v1:172.21.15.177:6789/0 conn(0x7f5bb809b2b0 legacy=0x7f5bb80af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb7fff640 1 -- 172.21.15.146:0/980211114 >> v1:172.21.15.146:6789/0 conn(0x7f5bb8160210 legacy=0x7f5bb806d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb7fff640 1 -- 172.21.15.146:0/980211114 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb806cb10 con 0x7f5bb80af840 2024-04-25T15:21:34.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.343+0000 7f5bb67fc640 1 -- 172.21.15.146:0/980211114 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732772786 0 0) 0x7f5bb806c550 con 0x7f5bb8160210 2024-04-25T15:21:34.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.344+0000 7f5bb7fff640 1 --2- 172.21.15.146:0/980211114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb816adc0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009e50 tx=0x7f5bb002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16531618f87edd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.344+0000 7f5bb67fc640 1 -- 172.21.15.146:0/980211114 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0036070 con 0x7f5bb80af840 2024-04-25T15:21:34.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.344+0000 7f5bb67fc640 1 -- 172.21.15.146:0/980211114 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bb0031600 con 0x7f5bb80af840 2024-04-25T15:21:34.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.344+0000 7f5bb67fc640 1 -- 172.21.15.146:0/980211114 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb00319d0 con 0x7f5bb80af840 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/980211114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 msgr2=0x7f5bb816adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/980211114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb816adc0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009e50 tx=0x7f5bb002a730 comp rx=0 tx=0).stop 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/980211114 shutdown_connections 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/980211114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb816adc0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/980211114 >> 172.21.15.146:0/980211114 conn(0x7f5bb8073a30 msgr2=0x7f5bb80b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/980211114 shutdown_connections 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.345+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/980211114 wait complete. 2024-04-25T15:21:34.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.347+0000 7f5bbdbbe640 1 Processor -- start 2024-04-25T15:21:34.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.347+0000 7f5bbdbbe640 1 -- start start 2024-04-25T15:21:34.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bbdbbe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bb809b2b0 0x7f5bb81606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bbdbbe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb8162d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bbdbbe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb8160210 0x7f5bb815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bbdbbe640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb806d0d0 con 0x7f5bb80af840 2024-04-25T15:21:34.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bbdbbe640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb806d440 con 0x7f5bb8160210 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bbdbbe640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb80afc50 con 0x7f5bb809b2b0 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bb6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb8162d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bb6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb8162d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40828/0 (socket says 172.21.15.146:40828) 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.348+0000 7f5bb6ffd640 1 -- 172.21.15.146:0/2221889534 learned_addr learned my addr 172.21.15.146:0/2221889534 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb7fff640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb8160210 0x7f5bb815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb77fe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bb809b2b0 0x7f5bb81606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb6ffd640 1 -- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bb809b2b0 msgr2=0x7f5bb81606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb6ffd640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bb809b2b0 0x7f5bb81606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb6ffd640 1 -- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb8160210 msgr2=0x7f5bb815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb6ffd640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb8160210 0x7f5bb815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb6ffd640 1 -- 172.21.15.146:0/2221889534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb8161170 con 0x7f5bb80af840 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb77fe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bb809b2b0 0x7f5bb81606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bb6ffd640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb8162d80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c460 tx=0x7f5bac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0052d0 con 0x7f5bb80af840 2024-04-25T15:21:34.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bac005470 con 0x7f5bb80af840 2024-04-25T15:21:34.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb8155f60 con 0x7f5bb80af840 2024-04-25T15:21:34.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac012870 con 0x7f5bb80af840 2024-04-25T15:21:34.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.349+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb8160bf0 con 0x7f5bb80af840 2024-04-25T15:21:34.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.350+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb806a0e0 con 0x7f5bb80af840 2024-04-25T15:21:34.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.351+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f5bac012a10 con 0x7f5bb80af840 2024-04-25T15:21:34.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.351+0000 7f5b9ffff640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ba4074540 0x7f5ba4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.351+0000 7f5bb77fe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ba4074540 0x7f5ba4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.351+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f5bac094f90 con 0x7f5bb80af840 2024-04-25T15:21:34.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.351+0000 7f5bb77fe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ba4074540 0x7f5ba4076a50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8000f60 tx=0x7f5ba8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.356+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5bac05f580 con 0x7f5bb80af840 2024-04-25T15:21:34.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.527+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5bb80b3df0 con 0x7f5bb80af840 2024-04-25T15:21:34.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.528+0000 7f5b9ffff640 1 -- 172.21.15.146:0/2221889534 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f5bac05ee80 con 0x7f5bb80af840 2024-04-25T15:21:34.530 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ba4074540 msgr2=0x7f5ba4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ba4074540 0x7f5ba4076a50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8000f60 tx=0x7f5ba8009040 comp rx=0 tx=0).stop 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 msgr2=0x7f5bb8162d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb8162d80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c460 tx=0x7f5bac00c920 comp rx=0 tx=0).stop 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 shutdown_connections 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ba4074540 0x7f5ba4076a50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb8160210 0x7f5bb815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb80af840 0x7f5bb8162d80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 --2- 172.21.15.146:0/2221889534 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5bb809b2b0 0x7f5bb81606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 >> 172.21.15.146:0/2221889534 conn(0x7f5bb8073a30 msgr2=0x7f5bb80b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 shutdown_connections 2024-04-25T15:21:34.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.530+0000 7f5bbdbbe640 1 -- 172.21.15.146:0/2221889534 wait complete. 2024-04-25T15:21:34.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:34.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.652+0000 7efbfc44a640 1 Processor -- start 2024-04-25T15:21:34.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.652+0000 7efbfc44a640 1 -- start start 2024-04-25T15:21:34.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.653+0000 7efbfc44a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efbf4071d30 0x7efbf40d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.653+0000 7efbfc44a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf40d3300 0x7efbf40cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbfc44a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbf40d2260 con 0x7efbf40b2cd0 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbfc44a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7efbf40d2540 con 0x7efbf4071d30 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbfc44a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efbf40d2820 con 0x7efbf40d3300 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efbf4071d30 0x7efbf40d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efbf4071d30 0x7efbf40d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40830/0 (socket says 172.21.15.146:40830) 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 -- 172.21.15.146:0/1164657186 learned_addr learned my addr 172.21.15.146:0/1164657186 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbfa9e7640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf40d3300 0x7efbf40cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 -- 172.21.15.146:0/1164657186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf40d3300 msgr2=0x7efbf40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf40d3300 0x7efbf40cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 -- 172.21.15.146:0/1164657186 >> v1:172.21.15.177:6789/0 conn(0x7efbf40b2cd0 legacy=0x7efbf4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbf99e5640 1 -- 172.21.15.146:0/1164657186 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf40d2b00 con 0x7efbf4071d30 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.654+0000 7efbfa9e7640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf40d3300 0x7efbf40cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.655+0000 7efbf99e5640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf4071d30 0x7efbf40d1c80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7efbe4009870 tx=0x7efbe402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faac0493fc14e20b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.655+0000 7efbf91e4640 1 -- 172.21.15.146:0/1164657186 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe4037070 con 0x7efbf4071d30 2024-04-25T15:21:34.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.655+0000 7efbf91e4640 1 -- 172.21.15.146:0/1164657186 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbe4032440 con 0x7efbf4071d30 2024-04-25T15:21:34.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.655+0000 7efbf91e4640 1 -- 172.21.15.146:0/1164657186 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe4032810 con 0x7efbf4071d30 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 -- 172.21.15.146:0/1164657186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf4071d30 msgr2=0x7efbf40d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf4071d30 0x7efbf40d1c80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7efbe4009870 tx=0x7efbe402a1c0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 -- 172.21.15.146:0/1164657186 shutdown_connections 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf40d3300 0x7efbf40cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 --2- 172.21.15.146:0/1164657186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf4071d30 0x7efbf40d1c80 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 -- 172.21.15.146:0/1164657186 >> 172.21.15.146:0/1164657186 conn(0x7efbf40b52e0 msgr2=0x7efbf40b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 -- 172.21.15.146:0/1164657186 shutdown_connections 2024-04-25T15:21:34.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.656+0000 7efbfc44a640 1 -- 172.21.15.146:0/1164657186 wait complete. 2024-04-25T15:21:34.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.658+0000 7efbfc44a640 1 Processor -- start 2024-04-25T15:21:34.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.658+0000 7efbfc44a640 1 -- start start 2024-04-25T15:21:34.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfc44a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efbf40b2cd0 0x7efbf40b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfc44a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 0x7efbf415e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfc44a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf41e81e0 0x7efbf40cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfc44a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efbf408dbf0 con 0x7efbf40d3300 2024-04-25T15:21:34.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfc44a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efbf40cfb70 con 0x7efbf40b2cd0 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfc44a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efbf408dd70 con 0x7efbf41e81e0 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 0x7efbf415e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 0x7efbf415e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40840/0 (socket says 172.21.15.146:40840) 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 -- 172.21.15.146:0/2555029988 learned_addr learned my addr 172.21.15.146:0/2555029988 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfa1e6640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efbf40b2cd0 0x7efbf40b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfa9e7640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf41e81e0 0x7efbf40cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 -- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf41e81e0 msgr2=0x7efbf40cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf41e81e0 0x7efbf40cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 -- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efbf40b2cd0 msgr2=0x7efbf40b1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efbf40b2cd0 0x7efbf40b1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbf99e5640 1 -- 172.21.15.146:0/2555029988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf41e88a0 con 0x7efbf40d3300 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.659+0000 7efbfa9e7640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf41e81e0 0x7efbf40cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.660+0000 7efbf99e5640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 0x7efbf415e160 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efbe402a200 tx=0x7efbe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.660+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe4041020 con 0x7efbf40d3300 2024-04-25T15:21:34.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.660+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbe40306c0 con 0x7efbf40d3300 2024-04-25T15:21:34.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.660+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf41e8b80 con 0x7efbf40d3300 2024-04-25T15:21:34.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.660+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe4037040 con 0x7efbf40d3300 2024-04-25T15:21:34.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.660+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf41e9140 con 0x7efbf40d3300 2024-04-25T15:21:34.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.661+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbf406a0e0 con 0x7efbf40d3300 2024-04-25T15:21:34.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.661+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7efbe4044020 con 0x7efbf40d3300 2024-04-25T15:21:34.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.662+0000 7efbeaffd640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efbd8074570 0x7efbd8076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.662+0000 7efbfa1e6640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efbd8074570 0x7efbd8076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.662+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7efbe40b5bb0 con 0x7efbf40d3300 2024-04-25T15:21:34.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.662+0000 7efbfa1e6640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efbd8074570 0x7efbd8076a80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efbf0007fd0 tx=0x7efbf0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.667+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efbe40bc080 con 0x7efbf40d3300 2024-04-25T15:21:34.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.844+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efbf41e8630 con 0x7efbf40d3300 2024-04-25T15:21:34.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.845+0000 7efbeaffd640 1 -- 172.21.15.146:0/2555029988 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7efbe404b030 con 0x7efbf40d3300 2024-04-25T15:21:34.847 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:34.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efbd8074570 msgr2=0x7efbd8076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efbd8074570 0x7efbd8076a80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efbf0007fd0 tx=0x7efbf0009040 comp rx=0 tx=0).stop 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 msgr2=0x7efbf415e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 0x7efbf415e160 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efbe402a200 tx=0x7efbe40046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 shutdown_connections 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efbd8074570 0x7efbd8076a80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efbf41e81e0 0x7efbf40cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf40d3300 0x7efbf415e160 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 --2- 172.21.15.146:0/2555029988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efbf40b2cd0 0x7efbf40b1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 >> 172.21.15.146:0/2555029988 conn(0x7efbf40b52e0 msgr2=0x7efbf4072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 shutdown_connections 2024-04-25T15:21:34.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.847+0000 7efbfc44a640 1 -- 172.21.15.146:0/2555029988 wait complete. 2024-04-25T15:21:34.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:34.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.970+0000 7fdb1e906640 1 Processor -- start 2024-04-25T15:21:34.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.970+0000 7fdb1e906640 1 -- start start 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb1e906640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb180b2cd0 0x7fdb180d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb1e906640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb180d2260 con 0x7fdb180d3300 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb1e906640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb180d2540 con 0x7fdb18071d30 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb1e906640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdb180d2820 con 0x7fdb180b2cd0 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb17fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb180b2cd0 0x7fdb180d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb177fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdb180d3300 0x7fdb18071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50194/0 (socket says 172.21.15.146:50194) 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb17fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb180b2cd0 0x7fdb180d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40848/0 (socket says 172.21.15.146:40848) 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb177fe640 1 -- 172.21.15.146:0/3592778089 learned_addr learned my addr 172.21.15.146:0/3592778089 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb17fff640 1 -- 172.21.15.146:0/3592778089 >> v1:172.21.15.177:6789/0 conn(0x7fdb18071d30 legacy=0x7fdb180cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb17fff640 1 -- 172.21.15.146:0/3592778089 >> v1:172.21.15.146:6789/0 conn(0x7fdb180d3300 legacy=0x7fdb18071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb17fff640 1 -- 172.21.15.146:0/3592778089 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb180d2b00 con 0x7fdb180b2cd0 2024-04-25T15:21:34.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.971+0000 7fdb16ffd640 1 -- 172.21.15.146:0/3592778089 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991178360 0 0) 0x7fdb180d2260 con 0x7fdb180d3300 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.972+0000 7fdb17fff640 1 --2- 172.21.15.146:0/3592778089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180d1c80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009790 tx=0x7fdb0802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d6edf677e7abd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.972+0000 7fdb16ffd640 1 -- 172.21.15.146:0/3592778089 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08037070 con 0x7fdb180b2cd0 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.972+0000 7fdb16ffd640 1 -- 172.21.15.146:0/3592778089 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0802ae40 con 0x7fdb180b2cd0 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.972+0000 7fdb16ffd640 1 -- 172.21.15.146:0/3592778089 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08032660 con 0x7fdb180b2cd0 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 -- 172.21.15.146:0/3592778089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 msgr2=0x7fdb180d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3592778089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180d1c80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009790 tx=0x7fdb0802a130 comp rx=0 tx=0).stop 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 -- 172.21.15.146:0/3592778089 shutdown_connections 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3592778089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180d1c80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 -- 172.21.15.146:0/3592778089 >> 172.21.15.146:0/3592778089 conn(0x7fdb180b52e0 msgr2=0x7fdb180b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 -- 172.21.15.146:0/3592778089 shutdown_connections 2024-04-25T15:21:34.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.973+0000 7fdb1e906640 1 -- 172.21.15.146:0/3592778089 wait complete. 2024-04-25T15:21:34.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.975+0000 7fdb1e906640 1 Processor -- start 2024-04-25T15:21:34.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.975+0000 7fdb1e906640 1 -- start start 2024-04-25T15:21:34.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb1e906640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb1e906640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d3300 0x7fdb181583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb1e906640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb1806ef20 0x7fdb1806f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb1e906640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1808dbf0 con 0x7fdb180b2cd0 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb1e906640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180cfb70 con 0x7fdb180d3300 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb1e906640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdb1808dd70 con 0x7fdb1806ef20 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb17fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb17fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40856/0 (socket says 172.21.15.146:40856) 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.976+0000 7fdb17fff640 1 -- 172.21.15.146:0/3058909237 learned_addr learned my addr 172.21.15.146:0/3058909237 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb177fe640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d3300 0x7fdb181583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb1cea3640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb1806ef20 0x7fdb1806f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb17fff640 1 -- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb1806ef20 msgr2=0x7fdb1806f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb17fff640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb1806ef20 0x7fdb1806f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb17fff640 1 -- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d3300 msgr2=0x7fdb181583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb17fff640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d3300 0x7fdb181583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb17fff640 1 -- 172.21.15.146:0/3058909237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1806f870 con 0x7fdb180b2cd0 2024-04-25T15:21:34.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb17fff640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180c9730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb08035040 tx=0x7fdb080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08042020 con 0x7fdb180b2cd0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb08005ba0 con 0x7fdb180b2cd0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb1815cac0 con 0x7fdb180b2cd0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08037040 con 0x7fdb180b2cd0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.977+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb180cac20 con 0x7fdb180b2cd0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.978+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb1806a0e0 con 0x7fdb180b2cd0 2024-04-25T15:21:34.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.979+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fdb08045020 con 0x7fdb180b2cd0 2024-04-25T15:21:34.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.979+0000 7fdb14ff9640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdb04074540 0x7fdb04076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:34.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.979+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fdb080b5b60 con 0x7fdb180b2cd0 2024-04-25T15:21:34.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.979+0000 7fdb177fe640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdb04074540 0x7fdb04076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:34.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.980+0000 7fdb177fe640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdb04074540 0x7fdb04076a50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdb181f92f0 tx=0x7fdb0c007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:34.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:34.984+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb0804c030 con 0x7fdb180b2cd0 2024-04-25T15:21:35.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.158+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdb18068680 con 0x7fdb180b2cd0 2024-04-25T15:21:35.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.158+0000 7fdb14ff9640 1 -- 172.21.15.146:0/3058909237 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7fdb1806f870 con 0x7fdb180b2cd0 2024-04-25T15:21:35.162 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdb04074540 msgr2=0x7fdb04076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdb04074540 0x7fdb04076a50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdb181f92f0 tx=0x7fdb0c007c00 comp rx=0 tx=0).stop 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 msgr2=0x7fdb180c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180c9730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb08035040 tx=0x7fdb080046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 shutdown_connections 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdb04074540 0x7fdb04076a50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb1806ef20 0x7fdb1806f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb180d3300 0x7fdb181583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 --2- 172.21.15.146:0/3058909237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb180b2cd0 0x7fdb180c9730 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.160+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 >> 172.21.15.146:0/3058909237 conn(0x7fdb180b52e0 msgr2=0x7fdb18072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.161+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 shutdown_connections 2024-04-25T15:21:35.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.161+0000 7fdb1e906640 1 -- 172.21.15.146:0/3058909237 wait complete. 2024-04-25T15:21:35.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:21:35.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.280+0000 7fcde8442640 1 Processor -- start 2024-04-25T15:21:35.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.280+0000 7fcde8442640 1 -- start start 2024-04-25T15:21:35.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde8442640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcde00cbc90 0x7fcde0153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde8442640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 0x7fcde015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde8442640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde00cf420 con 0x7fcde00d0ac0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde8442640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcde01541e0 con 0x7fcde00cbc90 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde8442640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcde00cfd10 con 0x7fcde00d01a0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde69df640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcde00d0ac0 0x7fcde00d0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50200/0 (socket says 172.21.15.146:50200) 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde69df640 1 -- 172.21.15.146:0/1524747046 learned_addr learned my addr 172.21.15.146:0/1524747046 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614486443 0 0) 0x7fcde00cf420 con 0x7fcde00d0ac0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde61de640 1 --2- 172.21.15.146:0/1524747046 >> v2:172.21.15.177:3300/0 conn(0x7fcde00cbc90 0x7fcde0153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdcc003520 con 0x7fcde00d0ac0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 0x7fcde015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4002858992 0 0) 0x7fcdcc003520 con 0x7fcde00d0ac0 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 msgr2=0x7fcde015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 0x7fcde015dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 >> v2:172.21.15.177:3300/0 conn(0x7fcde00cbc90 msgr2=0x7fcde0153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 --2- 172.21.15.146:0/1524747046 >> v2:172.21.15.177:3300/0 conn(0x7fcde00cbc90 0x7fcde0153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde00cc270 con 0x7fcde00d0ac0 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.282+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fcdd8001a10 con 0x7fcde00d0ac0 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 >> v1:172.21.15.146:6789/0 conn(0x7fcde00d0ac0 legacy=0x7fcde00d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 0x7fcdcc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcdcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdcc004d40 0x7fcdcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcde00cfd10 con 0x7fcdcc003f80 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdcc005280 con 0x7fcdcc004d40 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcdcc005480 con 0x7fcde00d01a0 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcdcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde69df640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdcc004d40 0x7fcdcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde61de640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 0x7fcdcc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde59dd640 1 -- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 msgr2=0x7fcdcc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:35.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde00d01a0 0x7fcdcc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde59dd640 1 -- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdcc004d40 msgr2=0x7fcdcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:35.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdcc004d40 0x7fcdcc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.283+0000 7fcde59dd640 1 -- 172.21.15.146:0/1524747046 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdcc003520 con 0x7fcdcc003f80 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.284+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcdcc004800 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009ac0 tx=0x7fcdcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceac6cb3f985b1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.284+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0036070 con 0x7fcdcc003f80 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.284+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd0031400 con 0x7fcdcc003f80 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.284+0000 7fcde51dc640 1 -- 172.21.15.146:0/1524747046 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0009690 con 0x7fcdcc003f80 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde8442640 1 -- 172.21.15.146:0/1524747046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 msgr2=0x7fcdcc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde8442640 1 --2- 172.21.15.146:0/1524747046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcdcc004800 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009ac0 tx=0x7fcdcc005830 comp rx=0 tx=0).stop 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde69df640 1 -- 172.21.15.146:0/1524747046 reap_dead start 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde8442640 1 -- 172.21.15.146:0/1524747046 shutdown_connections 2024-04-25T15:21:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde8442640 1 -- 172.21.15.146:0/1524747046 >> 172.21.15.146:0/1524747046 conn(0x7fcde00ce0c0 msgr2=0x7fcde00ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde8442640 1 -- 172.21.15.146:0/1524747046 shutdown_connections 2024-04-25T15:21:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.285+0000 7fcde8442640 1 -- 172.21.15.146:0/1524747046 wait complete. 2024-04-25T15:21:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.286+0000 7fcde8442640 1 Processor -- start 2024-04-25T15:21:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.287+0000 7fcde8442640 1 -- start start 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde8442640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcde0146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde8442640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde0154690 0x7fcde01f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde8442640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcde01f5620 0x7fcde01f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde8442640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcde00d05b0 con 0x7fcdcc003f80 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde8442640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcde00cf9a0 con 0x7fcde01f5620 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde8442640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcde00d0730 con 0x7fcde0154690 2024-04-25T15:21:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcde0146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcde0146110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40880/0 (socket says 172.21.15.146:40880) 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 -- 172.21.15.146:0/1341480033 learned_addr learned my addr 172.21.15.146:0/1341480033 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde69df640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcde01f5620 0x7fcde01f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde0154690 0x7fcde01f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 -- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde0154690 msgr2=0x7fcde01f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde0154690 0x7fcde01f50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 -- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcde01f5620 msgr2=0x7fcde01f5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcde01f5620 0x7fcde01f5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 -- 172.21.15.146:0/1341480033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde0154aa0 con 0x7fcdcc003f80 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde69df640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcde01f5620 0x7fcde01f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.288+0000 7fcde61de640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcde0146110 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0119b0 tx=0x7fcddc011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.289+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00faf0 con 0x7fcdcc003f80 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.289+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcddc00fc90 con 0x7fcdcc003f80 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.289+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc014440 con 0x7fcdcc003f80 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.289+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde01e72d0 con 0x7fcdcc003f80 2024-04-25T15:21:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.289+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde01e7890 con 0x7fcdcc003f80 2024-04-25T15:21:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.290+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcde006a0e0 con 0x7fcdcc003f80 2024-04-25T15:21:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.290+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fcddc012040 con 0x7fcdcc003f80 2024-04-25T15:21:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.291+0000 7fcdd6ffd640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcdc4074540 0x7fcdc4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.291+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fcddc095b30 con 0x7fcdcc003f80 2024-04-25T15:21:35.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.291+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcdc4074540 0x7fcdc4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:35.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.291+0000 7fcde59dd640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcdc4074540 0x7fcdc4076a50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0033000 tx=0x7fcdd0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.295+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcddc019030 con 0x7fcdcc003f80 2024-04-25T15:21:35.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.480+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcde00711c0 con 0x7fcdcc003f80 2024-04-25T15:21:35.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.480+0000 7fcdd6ffd640 1 -- 172.21.15.146:0/1341480033 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7fcddc0601b0 con 0x7fcdcc003f80 2024-04-25T15:21:35.482 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:21:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcdc4074540 msgr2=0x7fcdc4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcdc4074540 0x7fcdc4076a50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0033000 tx=0x7fcdd0008030 comp rx=0 tx=0).stop 2024-04-25T15:21:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 msgr2=0x7fcde0146110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:35.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcde0146110 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0119b0 tx=0x7fcddc011e70 comp rx=0 tx=0).stop 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 shutdown_connections 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcdc4074540 0x7fcdc4076a50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcde01f5620 0x7fcde01f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcde0154690 0x7fcde01f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 --2- 172.21.15.146:0/1341480033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdcc003f80 0x7fcde0146110 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 >> 172.21.15.146:0/1341480033 conn(0x7fcde00ce0c0 msgr2=0x7fcde015f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 shutdown_connections 2024-04-25T15:21:35.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:35.482+0000 7fcde8442640 1 -- 172.21.15.146:0/1341480033 wait complete. 2024-04-25T15:21:35.495 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:21:35.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:35.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:35.523 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[42086]: fuse finished with error 0 and tester_r 0 2024-04-25T15:21:35.525 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:21:41.627 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:21:41.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:41.627 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:21:41.646 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:41.647 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:41.665 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1449 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:21:41.666 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi146.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-04-25T15:21:41.666 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:21:41.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:41.666 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T15:21:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:21:41.723 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-25T15:21:41.723 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:21:41.723 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:41.741 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3659 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:21:41.741 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi177.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-04-25T15:21:41.741 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:21:41.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:41.742 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T15:21:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:21:41.799 INFO:teuthology.orchestra.run.smithi177.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-25T15:21:41.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:21:41.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.918+0000 7fe753f34640 1 Processor -- start 2024-04-25T15:21:41.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.918+0000 7fe753f34640 1 -- start start 2024-04-25T15:21:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe753f34640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe74c0d19d0 0x7fe74c0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe753f34640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe74c0b4a60 con 0x7fe74c0cfbe0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe753f34640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe74c0b4d40 con 0x7fe74c0cce10 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe753f34640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe74c0b5020 con 0x7fe74c0d19d0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe751cd0640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe74c0cfbe0 0x7fe74c0cf260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:51154/0 (socket says 172.21.15.146:51154) 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe751cd0640 1 -- 172.21.15.146:0/517672750 learned_addr learned my addr 172.21.15.146:0/517672750 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe7514cf640 1 --2- 172.21.15.146:0/517672750 >> v2:172.21.15.177:3300/0 conn(0x7fe74c0d19d0 0x7fe74c0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133817140 0 0) 0x7fe74c0b4a60 con 0x7fe74c0cfbe0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe738003520 con 0x7fe74c0cfbe0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3328779833 0 0) 0x7fe738003520 con 0x7fe74c0cfbe0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 >> v2:172.21.15.177:3300/0 conn(0x7fe74c0d19d0 msgr2=0x7fe74c0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 --2- 172.21.15.146:0/517672750 >> v2:172.21.15.177:3300/0 conn(0x7fe74c0d19d0 0x7fe74c0d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 >> v1:172.21.15.177:6789/0 conn(0x7fe74c0cce10 legacy=0x7fe74c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c0cead0 con 0x7fe74c0cfbe0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fe7480017f0 con 0x7fe74c0cfbe0 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.920+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 >> v1:172.21.15.146:6789/0 conn(0x7fe74c0cfbe0 legacy=0x7fe74c0cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:41.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe750cce640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7380039b0 0x7fe738003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe750cce640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe738004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe750cce640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7380051c0 0x7fe73800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c0b5020 con 0x7fe738004800 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe73800c160 con 0x7fe7380039b0 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe73800c360 con 0x7fe7380051c0 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7514cf640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe738004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7524d1640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7380051c0 0x7fe73800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe751cd0640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7380039b0 0x7fe738003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7514cf640 1 -- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7380051c0 msgr2=0x7fe73800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7514cf640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe7380051c0 0x7fe73800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7514cf640 1 -- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7380039b0 msgr2=0x7fe738003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7514cf640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7380039b0 0x7fe738003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7514cf640 1 -- 172.21.15.146:0/517672750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe738003520 con 0x7fe738004800 2024-04-25T15:21:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.921+0000 7fe7524d1640 1 -- 172.21.15.146:0/517672750 reap_dead start 2024-04-25T15:21:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.922+0000 7fe7514cf640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe738004c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009a90 tx=0x7fe738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff51a980975bda52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.922+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c036070 con 0x7fe738004800 2024-04-25T15:21:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.922+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73c031400 con 0x7fe738004800 2024-04-25T15:21:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.922+0000 7fe750cce640 1 -- 172.21.15.146:0/517672750 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c009690 con 0x7fe738004800 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 -- 172.21.15.146:0/517672750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 msgr2=0x7fe738004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe738004c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009a90 tx=0x7fe738005830 comp rx=0 tx=0).stop 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 -- 172.21.15.146:0/517672750 shutdown_connections 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 --2- 172.21.15.146:0/517672750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe738004c10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 -- 172.21.15.146:0/517672750 >> 172.21.15.146:0/517672750 conn(0x7fe74c0b73b0 msgr2=0x7fe74c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 -- 172.21.15.146:0/517672750 shutdown_connections 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.923+0000 7fe753f34640 1 -- 172.21.15.146:0/517672750 wait complete. 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.924+0000 7fe753f34640 1 Processor -- start 2024-04-25T15:21:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.925+0000 7fe753f34640 1 -- start start 2024-04-25T15:21:41.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe753f34640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe74c0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe753f34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c0b0110 0x7fe74c0b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe753f34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c0b0a60 0x7fe74c0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe753f34640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c0d1f60 con 0x7fe738004800 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe753f34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c0d1de0 con 0x7fe74c0b0110 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe753f34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe74c0d20e0 con 0x7fe74c0b0a60 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe74c0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe74c0c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40234/0 (socket says 172.21.15.146:40234) 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 -- 172.21.15.146:0/3267840801 learned_addr learned my addr 172.21.15.146:0/3267840801 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe7514cf640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c0b0110 0x7fe74c0b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe7524d1640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c0b0a60 0x7fe74c0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 -- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c0b0a60 msgr2=0x7fe74c0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c0b0a60 0x7fe74c0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 -- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c0b0110 msgr2=0x7fe74c0b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c0b0110 0x7fe74c0b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 -- 172.21.15.146:0/3267840801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c1f6090 con 0x7fe738004800 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe7524d1640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c0b0a60 0x7fe74c0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.926+0000 7fe751cd0640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe74c0c8d60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe74800e830 tx=0x7fe74800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.927+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7480070d0 con 0x7fe738004800 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.927+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe748018070 con 0x7fe738004800 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.927+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7480074b0 con 0x7fe738004800 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.927+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c0c78a0 con 0x7fe738004800 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.927+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c0af910 con 0x7fe738004800 2024-04-25T15:21:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.928+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c0c7b80 con 0x7fe738004800 2024-04-25T15:21:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.928+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fe7480057a0 con 0x7fe738004800 2024-04-25T15:21:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.929+0000 7fe7427fc640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe730074570 0x7fe730076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.929+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe7480953b0 con 0x7fe738004800 2024-04-25T15:21:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.929+0000 7fe7514cf640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe730074570 0x7fe730076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.929+0000 7fe7514cf640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe730074570 0x7fe730076a80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe73c033000 tx=0x7fe73c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:41.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:41.934+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe74805fa60 con 0x7fe738004800 2024-04-25T15:21:42.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.148+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe74c06a0e0 con 0x7fe738004800 2024-04-25T15:21:42.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.149+0000 7fe7427fc640 1 -- 172.21.15.146:0/3267840801 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fe74805f360 con 0x7fe738004800 2024-04-25T15:21:42.150 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:21:42.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.150+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe730074570 msgr2=0x7fe730076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.150+0000 7fe753f34640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe730074570 0x7fe730076a80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe73c033000 tx=0x7fe73c008030 comp rx=0 tx=0).stop 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.150+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 msgr2=0x7fe74c0c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.150+0000 7fe753f34640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe74c0c8d60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe74800e830 tx=0x7fe74800ecf0 comp rx=0 tx=0).stop 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 shutdown_connections 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe730074570 0x7fe730076a80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c0b0a60 0x7fe74c0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c0b0110 0x7fe74c0b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 --2- 172.21.15.146:0/3267840801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe738004800 0x7fe74c0c8d60 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 >> 172.21.15.146:0/3267840801 conn(0x7fe74c0b73b0 msgr2=0x7fe74c0cd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 shutdown_connections 2024-04-25T15:21:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42.151+0000 7fe753f34640 1 -- 172.21.15.146:0/3267840801 wait complete. 2024-04-25T15:21:42.163 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:21:42.163 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:21:42.164 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:21:42.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.164 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:21:42.179 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:21:42.179 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:21:42.179 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:42.180 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42659sec preferred_lft 42659sec 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.181 INFO:teuthology.orchestra.run.smithi146.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:42.182 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:42.182 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7820:1ff:feb4:c7f1/64 scope link 2024-04-25T15:21:42.182 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:42.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.182 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:21:42.242 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:21:42.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.244 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:21:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:21:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:21:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:21:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:21:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:21:42.300 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:21:42.300 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:21:42.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:21:42.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:21:42.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:21:42.300 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:21:42.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:21:42.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:42.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.425 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:21:42.439 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:21:42.439 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:21:42.440 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:21:42.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.440 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-25T15:21:42.504 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T15:21:42.504 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:42.504 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:42.559 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:21:42.560 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:42.560 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:42.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:21:42.680 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:42.736 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:21:42.736 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:21:42.736 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:21:42.736 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:21:42.736 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:21:42.736 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:42.737 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:21:42.738 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:42.739 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:42.739 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:42.739 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:42.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.740 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:21:42.795 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:21:42.795 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f) 2024-04-25T15:21:42.838 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:21:42.861 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:21:42.920 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:21:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:21:42.922 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:21:42.922 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:21:42.922 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:42.922 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:42.922 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:42.922 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:42.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:42.923 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:21:42.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:2024-04-25T15:21:42.926+0000 7f2a3b5ce480 -1 init, newargv = 0x7f2a2c052070 newargc=15 2024-04-25T15:21:42.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[53345]: starting ceph client 2024-04-25T15:21:42.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[53345]: starting fuse 2024-04-25T15:21:42.942 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:21:42.943 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T15:21:42.943 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.0")) 2024-04-25T15:21:42.944 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:21:43.061 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.0.53345.asok 2024-04-25T15:21:43.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:43.071 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53345.asok 2024-04-25T15:21:43.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:43.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53345.asok status 2024-04-25T15:21:43.212 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:43.212 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:21:43.212 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:21:43.212 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "0", 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "53345", 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f" 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5967, 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 5967 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:43.213 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1190023908 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1190023908 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.5967 192.168.0.1:0/1190023908", 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/1190023908", 2024-04-25T15:21:43.214 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 5, 2024-04-25T15:21:43.215 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 17, 2024-04-25T15:21:43.215 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 41, 2024-04-25T15:21:43.215 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:21:43.215 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:21:43.215 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:21:43.215 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:43.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:21:43.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.328+0000 7fad56d34640 1 Processor -- start 2024-04-25T15:21:43.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.328+0000 7fad56d34640 1 -- start start 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad56d34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad500d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad56d34640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad500d0020 0x7fad500d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad56d34640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad500b9200 con 0x7fad50163d80 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad56d34640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fad5006c550 con 0x7fad500d0020 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad56d34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fad5006c830 con 0x7fad500d4f60 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad552d1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fad50163d80 0x7fad50164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:51162/0 (socket says 172.21.15.146:51162) 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad552d1640 1 -- 172.21.15.146:0/2485478637 learned_addr learned my addr 172.21.15.146:0/2485478637 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254490980 0 0) 0x7fad48001490 con 0x7fad50163d80 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad40003520 con 0x7fad50163d80 2024-04-25T15:21:43.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad500d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad4ffff640 1 --2- 172.21.15.146:0/2485478637 >> v2:172.21.15.177:3300/0 conn(0x7fad500d0020 0x7fad500d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 119463113 0 0) 0x7fad40003520 con 0x7fad50163d80 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 msgr2=0x7fad500d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.330+0000 7fad4f7fe640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad500d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 >> v2:172.21.15.177:3300/0 conn(0x7fad500d0020 msgr2=0x7fad500d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 --2- 172.21.15.146:0/2485478637 >> v2:172.21.15.177:3300/0 conn(0x7fad500d0020 0x7fad500d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad500d0970 con 0x7fad50163d80 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fad48001bc0 con 0x7fad50163d80 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 >> v1:172.21.15.146:6789/0 conn(0x7fad50163d80 legacy=0x7fad50164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad400039b0 0x7fad40003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad40004d40 0x7fad40005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fad5006c550 con 0x7fad400039b0 2024-04-25T15:21:43.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad5006c830 con 0x7fad40004d40 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fad4000baf0 con 0x7fad500d4f60 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad400039b0 0x7fad40003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad552d1640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad40004d40 0x7fad40005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4ffff640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad54ad0640 1 -- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 msgr2=0x7fad40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad54ad0640 1 -- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad40004d40 msgr2=0x7fad40005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad40004d40 0x7fad40005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad54ad0640 1 -- 172.21.15.146:0/2485478637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad40003520 con 0x7fad400039b0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.331+0000 7fad4ffff640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad500d4f60 0x7fad40004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.332+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad400039b0 0x7fad40003e20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fad50163ce0 tx=0x7fad40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1868ed9d3feca2b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.332+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44036070 con 0x7fad400039b0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.332+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad44031400 con 0x7fad400039b0 2024-04-25T15:21:43.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.332+0000 7fad4f7fe640 1 -- 172.21.15.146:0/2485478637 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44009690 con 0x7fad400039b0 2024-04-25T15:21:43.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad56d34640 1 -- 172.21.15.146:0/2485478637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad400039b0 msgr2=0x7fad40003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:43.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad56d34640 1 --2- 172.21.15.146:0/2485478637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad400039b0 0x7fad40003e20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fad50163ce0 tx=0x7fad40005830 comp rx=0 tx=0).stop 2024-04-25T15:21:43.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad552d1640 1 -- 172.21.15.146:0/2485478637 reap_dead start 2024-04-25T15:21:43.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad56d34640 1 -- 172.21.15.146:0/2485478637 shutdown_connections 2024-04-25T15:21:43.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad56d34640 1 -- 172.21.15.146:0/2485478637 >> 172.21.15.146:0/2485478637 conn(0x7fad500d2450 msgr2=0x7fad500d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:43.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad56d34640 1 -- 172.21.15.146:0/2485478637 shutdown_connections 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.333+0000 7fad56d34640 1 -- 172.21.15.146:0/2485478637 wait complete. 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.335+0000 7fad56d34640 1 Processor -- start 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.335+0000 7fad56d34640 1 -- start start 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad56d34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad400039b0 0x7fad50162d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad56d34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad5014ada0 0x7fad50154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad56d34640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 0x7fad501564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad56d34640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fad500d15f0 con 0x7fad501560d0 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad56d34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad500d1070 con 0x7fad400039b0 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad56d34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fad500b5150 con 0x7fad5014ada0 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad552d1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 0x7fad501564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad552d1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 0x7fad501564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40250/0 (socket says 172.21.15.146:40250) 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad552d1640 1 -- 172.21.15.146:0/2784933562 learned_addr learned my addr 172.21.15.146:0/2784933562 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad400039b0 0x7fad50162d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.336+0000 7fad4ffff640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad5014ada0 0x7fad50154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad552d1640 1 -- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad5014ada0 msgr2=0x7fad50154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad552d1640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad5014ada0 0x7fad50154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad552d1640 1 -- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad400039b0 msgr2=0x7fad50162d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad552d1640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad400039b0 0x7fad50162d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad552d1640 1 -- 172.21.15.146:0/2784933562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad50156d30 con 0x7fad501560d0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad552d1640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 0x7fad501564e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fad480013d0 tx=0x7fad48004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48019070 con 0x7fad501560d0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad480078c0 con 0x7fad501560d0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad50157010 con 0x7fad501560d0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480133f0 con 0x7fad501560d0 2024-04-25T15:21:43.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.337+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad501575d0 con 0x7fad501560d0 2024-04-25T15:21:43.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.338+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad501579b0 con 0x7fad501560d0 2024-04-25T15:21:43.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.339+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fad4801c070 con 0x7fad501560d0 2024-04-25T15:21:43.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.339+0000 7fad4d7fa640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fad34074570 0x7fad34076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:43.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.339+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fad48094e00 con 0x7fad501560d0 2024-04-25T15:21:43.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.339+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fad34074570 0x7fad34076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:43.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.340+0000 7fad54ad0640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fad34074570 0x7fad34076a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fad44033000 tx=0x7fad44008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:43.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.344+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad501579b0 con 0x7fad501560d0 2024-04-25T15:21:43.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.563+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fad5006a0e0 con 0x7fad501560d0 2024-04-25T15:21:43.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.563+0000 7fad4d7fa640 1 -- 172.21.15.146:0/2784933562 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fad500d0970 con 0x7fad501560d0 2024-04-25T15:21:43.565 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:21:43.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fad34074570 msgr2=0x7fad34076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:43.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fad34074570 0x7fad34076a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fad44033000 tx=0x7fad44008030 comp rx=0 tx=0).stop 2024-04-25T15:21:43.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 msgr2=0x7fad501564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:43.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 0x7fad501564e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fad480013d0 tx=0x7fad48004ed0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 shutdown_connections 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fad34074570 0x7fad34076a80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad501560d0 0x7fad501564e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad5014ada0 0x7fad50154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 --2- 172.21.15.146:0/2784933562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad400039b0 0x7fad50162d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 >> 172.21.15.146:0/2784933562 conn(0x7fad500d2450 msgr2=0x7fad500b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 shutdown_connections 2024-04-25T15:21:43.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:43.565+0000 7fad56d34640 1 -- 172.21.15.146:0/2784933562 wait complete. 2024-04-25T15:21:43.578 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:21:43.578 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:21:43.578 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:21:43.578 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:21:43.578 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:43.578 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:21:43.579 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f 2024-04-25T15:21:43.579 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:21:43.579 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:21:43.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:43.579 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:43.598 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42630sec preferred_lft 42630sec 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.599 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::7cc7:d2ff:fef5:71b6/64 scope link 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:43.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:43.600 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:21:43.664 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:21:43.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:43.666 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:21:43.723 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:21:43.724 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:21:43.724 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:21:43.724 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:21:43.724 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:21:43.724 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:21:43.724 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:21:43.724 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:21:43.724 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:21:43.724 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:21:43.724 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:21:43.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:21:43.847 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:43.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:43.854 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:21:43.870 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:21:43.870 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:21:43.870 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:21:43.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:43.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-04-25T15:21:43.933 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-25T15:21:43.934 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:43.934 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:43.988 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:21:43.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:43.989 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:44.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:21:44.106 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:44.162 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:21:44.162 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:21:44.162 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:21:44.163 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:21:44.164 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:21:44.165 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:44.165 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:44.165 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:44.165 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:44.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:44.165 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:21:44.218 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:21:44.218 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f) 2024-04-25T15:21:44.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:21:44.283 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:21:44.340 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:44.342 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:44.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:44.343 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:21:44.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:2024-04-25T15:21:44.348+0000 7f508840b480 -1 init, newargv = 0x7f506c001290 newargc=15 2024-04-25T15:21:44.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[42562]: starting ceph client 2024-04-25T15:21:44.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[42562]: starting fuse 2024-04-25T15:21:44.364 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:21:44.364 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:21:44.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:21:44.364 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:21:44.364 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:21:44.364 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:21:44.365 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.1")) 2024-04-25T15:21:44.366 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:21:44.480 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.1.42562.asok 2024-04-25T15:21:44.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:44.491 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42562.asok 2024-04-25T15:21:44.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:44.491 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42562.asok status 2024-04-25T15:21:44.631 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:21:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:21:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:21:44.631 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "1", 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "42562", 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f" 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 5189, 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:21:44.632 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 5189 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 3062561742 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 3062561742 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:44.633 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.5189 192.168.0.1:0/3062561742", 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.1:0/3062561742", 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 5, 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 17, 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 41, 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:21:44.634 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:21:44.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:21:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.759+0000 7f1781db2640 1 Processor -- start 2024-04-25T15:21:44.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.759+0000 7f1781db2640 1 -- start start 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f1781db2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f1781db2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f177c0cfa90 0x7f177c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f1781db2640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f177c0cfea0 con 0x7f177c0d2c50 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f1781db2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f177c0b4f20 con 0x7f177c0cfa90 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f1781db2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f177c0b3a70 con 0x7f177c0b4b10 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177affd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f177c0cfa90 0x7f177c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177bfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f177c0d2c50 0x7f177c0af490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:35888/0 (socket says 172.21.15.146:35888) 2024-04-25T15:21:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177bfff640 1 -- 172.21.15.146:0/508424720 learned_addr learned my addr 172.21.15.146:0/508424720 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177affd640 1 -- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 msgr2=0x7f177c0cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177affd640 1 --2- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c0cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177affd640 1 -- 172.21.15.146:0/508424720 >> v1:172.21.15.177:6789/0 conn(0x7f177c0d2c50 legacy=0x7f177c0af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177affd640 1 -- 172.21.15.146:0/508424720 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c0b3d50 con 0x7f177c0cfa90 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.761+0000 7f177b7fe640 1 --2- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c0cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.762+0000 7f177affd640 1 --2- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0d2710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f17700097a0 tx=0x7f177002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef75aac76e6b3bee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.762+0000 7f177a7fc640 1 -- 172.21.15.146:0/508424720 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770037070 con 0x7f177c0cfa90 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.762+0000 7f177a7fc640 1 -- 172.21.15.146:0/508424720 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f177002ae50 con 0x7f177c0cfa90 2024-04-25T15:21:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.762+0000 7f177a7fc640 1 -- 172.21.15.146:0/508424720 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770032660 con 0x7f177c0cfa90 2024-04-25T15:21:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 -- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 msgr2=0x7f177c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 --2- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0d2710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f17700097a0 tx=0x7f177002a100 comp rx=0 tx=0).stop 2024-04-25T15:21:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 -- 172.21.15.146:0/508424720 shutdown_connections 2024-04-25T15:21:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 --2- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0d2710 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 --2- 172.21.15.146:0/508424720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c0cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 -- 172.21.15.146:0/508424720 >> 172.21.15.146:0/508424720 conn(0x7f177c0b73b0 msgr2=0x7f177c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 -- 172.21.15.146:0/508424720 shutdown_connections 2024-04-25T15:21:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.763+0000 7f1781db2640 1 -- 172.21.15.146:0/508424720 wait complete. 2024-04-25T15:21:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.765+0000 7f1781db2640 1 Processor -- start 2024-04-25T15:21:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.765+0000 7f1781db2640 1 -- start start 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1781db2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1781db2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1781db2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f177c0d2c50 0x7f177c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1781db2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f177c0b45f0 con 0x7f177c0d2c50 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1781db2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f177c0b4310 con 0x7f177c0cfa90 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1781db2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f177c0d3060 con 0x7f177c0b4b10 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f177c0d2c50 0x7f177c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f177c0d2c50 0x7f177c0b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40262/0 (socket says 172.21.15.146:40262) 2024-04-25T15:21:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177bfff640 1 -- 172.21.15.146:0/1265926465 learned_addr learned my addr 172.21.15.146:0/1265926465 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 -- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 msgr2=0x7f177c0caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 -- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f177c0d2c50 msgr2=0x7f177c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f177c0d2c50 0x7f177c0b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 -- 172.21.15.146:0/1265926465 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c1f4550 con 0x7f177c0b4b10 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f177b7fe640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c153b70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e820 tx=0x7f176c00ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.767+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c005990 con 0x7f177c0b4b10 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.768+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176c005b30 con 0x7f177c0b4b10 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.768+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177c1f4830 con 0x7f177c0b4b10 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.768+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c00c940 con 0x7f177c0b4b10 2024-04-25T15:21:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.768+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f177c1f4df0 con 0x7f177c0b4b10 2024-04-25T15:21:44.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.768+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f177c06a0e0 con 0x7f177c0b4b10 2024-04-25T15:21:44.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.769+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f176c00cae0 con 0x7f177c0b4b10 2024-04-25T15:21:44.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.770+0000 7f1763fff640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1768074540 0x7f1768076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:44.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.770+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f176c095550 con 0x7f177c0b4b10 2024-04-25T15:21:44.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.770+0000 7f177affd640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1768074540 0x7f1768076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:44.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.770+0000 7f177affd640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1768074540 0x7f1768076a50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1770034000 tx=0x7f177002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:44.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.775+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f176c017020 con 0x7f177c0b4b10 2024-04-25T15:21:44.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.987+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f177c068580 con 0x7f177c0b4b10 2024-04-25T15:21:44.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.987+0000 7f1763fff640 1 -- 172.21.15.146:0/1265926465 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f176c05f610 con 0x7f177c0b4b10 2024-04-25T15:21:44.989 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1768074540 msgr2=0x7f1768076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1768074540 0x7f1768076a50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1770034000 tx=0x7f177002e000 comp rx=0 tx=0).stop 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 msgr2=0x7f177c153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c153b70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e820 tx=0x7f176c00ece0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 shutdown_connections 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1768074540 0x7f1768076a50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f177c0d2c50 0x7f177c0b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f177c0cfa90 0x7f177c0caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 --2- 172.21.15.146:0/1265926465 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f177c0b4b10 0x7f177c153b70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 >> 172.21.15.146:0/1265926465 conn(0x7f177c0b73b0 msgr2=0x7f177c06d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 shutdown_connections 2024-04-25T15:21:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:44.989+0000 7f1781db2640 1 -- 172.21.15.146:0/1265926465 wait complete. 2024-04-25T15:21:45.002 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:21:45.002 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:21:45.002 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:21:45.002 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T15:21:45.002 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T15:21:45.003 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:21:45.003 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f 2024-04-25T15:21:45.003 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:21:45.003 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:21:45.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.003 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:21:45.018 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:45.019 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:45.020 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42656sec preferred_lft 42656sec 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link 2024-04-25T15:21:45.021 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.022 INFO:teuthology.orchestra.run.smithi146.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:45.022 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:45.022 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7820:1ff:feb4:c7f1/64 scope link 2024-04-25T15:21:45.022 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:45.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.022 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:21:45.081 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:21:45.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.083 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:21:45.138 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:21:45.139 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:21:45.139 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:21:45.139 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:21:45.139 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:21:45.139 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:21:45.139 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:21:45.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:21:45.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:21:45.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:21:45.139 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:21:45.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:21:45.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:45.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.262 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:21:45.276 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:21:45.277 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:21:45.277 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:21:45.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.1 up 2024-04-25T15:21:45.340 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-25T15:21:45.340 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:45.340 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T15:21:45.396 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:21:45.396 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:45.396 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T15:21:45.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:21:45.515 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:45.571 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:21:45.571 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:21:45.571 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:21:45.571 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:21:45.571 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:21:45.571 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:45.572 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:45.573 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:45.574 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:45.574 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:45.574 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:45.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.574 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:21:45.626 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:21:45.626 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-25T15:21:45.627 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f) 2024-04-25T15:21:45.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:21:45.691 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:45.748 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:45.749 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:21:45.750 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:21:45.751 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:45.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.752 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:21:45.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:2024-04-25T15:21:45.757+0000 7f5989a19480 -1 init, newargv = 0x5630caf06390 newargc=15 2024-04-25T15:21:45.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[53771]: starting ceph client 2024-04-25T15:21:45.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[53771]: starting fuse 2024-04-25T15:21:45.774 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:21:45.774 INFO:teuthology.orchestra.run.smithi146.stdout:44 2024-04-25T15:21:45.774 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-25T15:21:45.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:21:45.775 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:21:45.776 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:21:45.777 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.2")) 2024-04-25T15:21:45.777 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:21:45.890 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.2.53771.asok 2024-04-25T15:21:45.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:45.900 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.53771.asok 2024-04-25T15:21:45.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:45.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.53771.asok status 2024-04-25T15:21:46.041 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "2", 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "53771", 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f" 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:21:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5985, 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 5985 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 5174689 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:21:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 5174689 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.5985 192.168.0.2:0/5174689", 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.2:0/5174689", 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 5, 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 17, 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 41, 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:21:46.044 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:46.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:21:46.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:21:46.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:21:46.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.160+0000 7f6aa5b46640 1 Processor -- start 2024-04-25T15:21:46.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.160+0000 7f6aa5b46640 1 -- start start 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6aa5b46640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6aa015ded0 0x7f6aa00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6aa5b46640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa009c2a0 con 0x7f6aa00d21f0 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6aa5b46640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa00b4020 con 0x7f6aa00b3c10 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6aa5b46640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa00d1d60 con 0x7f6aa015ded0 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6a97fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6aa015ded0 0x7f6aa00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6a9f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6aa00d21f0 0x7f6aa006f2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:51178/0 (socket says 172.21.15.146:51178) 2024-04-25T15:21:46.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.161+0000 7f6a9f7fe640 1 -- 172.21.15.146:0/1408270045 learned_addr learned my addr 172.21.15.146:0/1408270045 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:46.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.162+0000 7f6a97fff640 1 -- 172.21.15.146:0/1408270045 >> v1:172.21.15.177:6789/0 conn(0x7f6aa00b3c10 legacy=0x7f6aa00b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.162+0000 7f6a97fff640 1 -- 172.21.15.146:0/1408270045 >> v1:172.21.15.146:6789/0 conn(0x7f6aa00d21f0 legacy=0x7f6aa006f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.162+0000 7f6a97fff640 1 -- 172.21.15.146:0/1408270045 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa009bce0 con 0x7f6aa015ded0 2024-04-25T15:21:46.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.162+0000 7f6a97fff640 1 --2- 172.21.15.146:0/1408270045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa015ded0 0x7f6aa00b36d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6a880097a0 tx=0x7f6a8802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08bb88da7d71765 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:46.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6a9effd640 1 -- 172.21.15.146:0/1408270045 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88037070 con 0x7f6aa015ded0 2024-04-25T15:21:46.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6a9effd640 1 -- 172.21.15.146:0/1408270045 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a88032640 con 0x7f6aa015ded0 2024-04-25T15:21:46.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6a9effd640 1 -- 172.21.15.146:0/1408270045 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a88032a10 con 0x7f6aa015ded0 2024-04-25T15:21:46.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 -- 172.21.15.146:0/1408270045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa015ded0 msgr2=0x7f6aa00b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/1408270045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa015ded0 0x7f6aa00b36d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6a880097a0 tx=0x7f6a8802a2c0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 -- 172.21.15.146:0/1408270045 shutdown_connections 2024-04-25T15:21:46.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/1408270045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa015ded0 0x7f6aa00b36d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 -- 172.21.15.146:0/1408270045 >> 172.21.15.146:0/1408270045 conn(0x7f6aa00b7530 msgr2=0x7f6aa00b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:46.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 -- 172.21.15.146:0/1408270045 shutdown_connections 2024-04-25T15:21:46.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.163+0000 7f6aa5b46640 1 -- 172.21.15.146:0/1408270045 wait complete. 2024-04-25T15:21:46.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.165+0000 7f6aa5b46640 1 Processor -- start 2024-04-25T15:21:46.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.165+0000 7f6aa5b46640 1 -- start start 2024-04-25T15:21:46.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.166+0000 7f6aa5b46640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 0x7f6aa00cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:46.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.166+0000 7f6aa5b46640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aa00d21f0 0x7f6aa015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:46.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.166+0000 7f6aa5b46640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa015ded0 0x7f6aa015be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.166+0000 7f6aa5b46640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa015e2e0 con 0x7f6aa00b3c10 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.166+0000 7f6aa5b46640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa00d19f0 con 0x7f6aa015ded0 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.166+0000 7f6aa5b46640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa009b400 con 0x7f6aa00d21f0 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 0x7f6aa00cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 0x7f6aa00cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40284/0 (socket says 172.21.15.146:40284) 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 -- 172.21.15.146:0/3866762691 learned_addr learned my addr 172.21.15.146:0/3866762691 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a97fff640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aa00d21f0 0x7f6aa015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9ffff640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa015ded0 0x7f6aa015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 -- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aa00d21f0 msgr2=0x7f6aa015b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aa00d21f0 0x7f6aa015b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 -- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa015ded0 msgr2=0x7f6aa015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa015ded0 0x7f6aa015be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 -- 172.21.15.146:0/3866762691 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa015e820 con 0x7f6aa00b3c10 2024-04-25T15:21:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9f7fe640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 0x7f6aa00cb9e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000c480 tx=0x7f6a9000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:46.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a90017070 con 0x7f6aa00b3c10 2024-04-25T15:21:46.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a90005ce0 con 0x7f6aa00b3c10 2024-04-25T15:21:46.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a900072f0 con 0x7f6aa00b3c10 2024-04-25T15:21:46.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa00cc470 con 0x7f6aa00b3c10 2024-04-25T15:21:46.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.167+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa00c9e20 con 0x7f6aa00b3c10 2024-04-25T15:21:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.168+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa006a0e0 con 0x7f6aa00b3c10 2024-04-25T15:21:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.169+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f6a9001a070 con 0x7f6aa00b3c10 2024-04-25T15:21:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.170+0000 7f6a9cff9640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6a80074540 0x7f6a80076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.170+0000 7f6a97fff640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6a80074540 0x7f6a80076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.170+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f6a90094580 con 0x7f6aa00b3c10 2024-04-25T15:21:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.170+0000 7f6a97fff640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6a80074540 0x7f6a80076a50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6a88034000 tx=0x7f6a8802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:46.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.174+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6aa015e820 con 0x7f6aa00b3c10 2024-04-25T15:21:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.386+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6aa00cd3f0 con 0x7f6aa00b3c10 2024-04-25T15:21:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.386+0000 7f6a9cff9640 1 -- 172.21.15.146:0/3866762691 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f6a9005e640 con 0x7f6aa00b3c10 2024-04-25T15:21:46.388 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6a80074540 msgr2=0x7f6a80076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6a80074540 0x7f6a80076a50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6a88034000 tx=0x7f6a8802e000 comp rx=0 tx=0).stop 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 msgr2=0x7f6aa00cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 0x7f6aa00cb9e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6a9000c480 tx=0x7f6a9000c940 comp rx=0 tx=0).stop 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 shutdown_connections 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6a80074540 0x7f6a80076a50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aa015ded0 0x7f6aa015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aa00d21f0 0x7f6aa015b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 --2- 172.21.15.146:0/3866762691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aa00b3c10 0x7f6aa00cb9e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 >> 172.21.15.146:0/3866762691 conn(0x7f6aa00b7530 msgr2=0x7f6aa00af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 shutdown_connections 2024-04-25T15:21:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:46.388+0000 7f6aa5b46640 1 -- 172.21.15.146:0/3866762691 wait complete. 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:21:46.401 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:21:46.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:46.402 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:21:46.420 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:21:46.420 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:21:46.420 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:21:46.420 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.420 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42628sec preferred_lft 42628sec 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:21:46.421 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::7cc7:d2ff:fef5:71b6/64 scope link 2024-04-25T15:21:46.422 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:21:46.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:46.423 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:21:46.487 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:21:46.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:46.489 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:21:46.548 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:21:46.548 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:21:46.548 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:21:46.548 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:21:46.548 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:21:46.548 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:21:46.548 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:21:46.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:21:46.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:21:46.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:21:46.549 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:21:46.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:21:46.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:46.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:46.681 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:21:46.696 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:21:46.696 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:21:46.696 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:21:46.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:46.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.1 up 2024-04-25T15:21:46.763 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-25T15:21:46.763 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:46.763 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T15:21:46.818 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:21:46.818 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:21:46.818 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T15:21:46.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:21:46.937 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:21:46.993 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:46.994 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:21:46.995 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:46.996 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:46.996 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:46.996 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:46.996 INFO:teuthology.orchestra.run.smithi177.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:46.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:46.996 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:21:47.048 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:21:47.048 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-25T15:21:47.048 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f) 2024-04-25T15:21:47.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:21:47.114 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:21:47.171 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:21:47.171 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:21:47.171 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:21:47.171 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:21:47.171 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:21:47.171 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:21:47.172 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:47.173 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:21:47.174 INFO:teuthology.orchestra.run.smithi177.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:47.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:47.174 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:21:47.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:2024-04-25T15:21:47.179+0000 7f1545f05480 -1 init, newargv = 0x7f1538001610 newargc=15 2024-04-25T15:21:47.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[42921]: starting ceph client 2024-04-25T15:21:47.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[42921]: starting fuse 2024-04-25T15:21:47.195 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:21:47.195 INFO:teuthology.orchestra.run.smithi177.stdout:44 2024-04-25T15:21:47.195 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-25T15:21:47.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:21:47.196 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.3")) 2024-04-25T15:21:47.197 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:21:47.313 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.3.42921.asok 2024-04-25T15:21:47.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:21:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:47.325 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.42921.asok 2024-04-25T15:21:47.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:47.325 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.42921.asok status 2024-04-25T15:21:47.475 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "3", 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "42921", 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f" 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 5204, 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:21:47.476 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 5204 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 990225218 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 990225218 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.5204 192.168.0.2:0/990225218", 2024-04-25T15:21:47.477 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.2:0/990225218", 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 5, 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 17, 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 41, 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:21:47.478 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:21:47.484 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'touch file') 2024-04-25T15:21:47.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:21:47.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:21:47.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:47.558 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quiesce.block -v 1 file) 2024-04-25T15:21:47.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 quiesce path /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f --wait 2024-04-25T15:21:47.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.735+0000 7f6df75c5640 1 Processor -- start 2024-04-25T15:21:47.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.735+0000 7f6df75c5640 1 -- start start 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df75c5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6df00ce900 0x7f6df00d2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df75c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 0x7f6df0167cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df75c5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df00cef00 con 0x7f6df00d2840 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df75c5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f6df00cff80 con 0x7f6df00ce900 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df75c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6df00cf1c0 con 0x7f6df01658b0 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df5361640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6df00d2840 0x7f6df00ce1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:51182/0 (socket says 172.21.15.146:51182) 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.736+0000 7f6df5361640 1 -- 172.21.15.146:0/604933306 learned_addr learned my addr 172.21.15.146:0/604933306 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6df4b60640 1 --2- 172.21.15.146:0/604933306 >> v2:172.21.15.177:3300/0 conn(0x7f6df00ce900 0x7f6df00d2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6df5b62640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 0x7f6df0167cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676034284 0 0) 0x7f6df00cef00 con 0x7f6df00d2840 2024-04-25T15:21:47.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ddc003520 con 0x7f6df00d2840 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1004160213 0 0) 0x7f6ddc003520 con 0x7f6df00d2840 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 msgr2=0x7f6df0167cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 0x7f6df0167cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 >> v2:172.21.15.177:3300/0 conn(0x7f6df00ce900 msgr2=0x7f6df00d2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 --2- 172.21.15.146:0/604933306 >> v2:172.21.15.177:3300/0 conn(0x7f6df00ce900 0x7f6df00d2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df00cf4a0 con 0x7f6df00d2840 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f6dec001670 con 0x7f6df00d2840 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 >> v1:172.21.15.146:6789/0 conn(0x7f6df00d2840 legacy=0x7f6df00ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 0x7f6ddc0039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6ddc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ddc004d40 0x7f6ddc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6df00cff80 con 0x7f6ddc003f50 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6df00cf1c0 con 0x7f6ddc004d40 2024-04-25T15:21:47.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc00ba40 con 0x7f6df01658b0 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6df4b60640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6ddc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6df5361640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 0x7f6ddc0039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.737+0000 7f6df5b62640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ddc004d40 0x7f6ddc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6df4b60640 1 -- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 msgr2=0x7f6ddc0039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6df4b60640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df01658b0 0x7f6ddc0039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6df4b60640 1 -- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ddc004d40 msgr2=0x7f6ddc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6df4b60640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ddc004d40 0x7f6ddc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6df4b60640 1 -- 172.21.15.146:0/604933306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc003520 con 0x7f6ddc003f50 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6df4b60640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6ddc004800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009ac0 tx=0x7f6ddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d089a246b655d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0036070 con 0x7f6ddc003f50 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de0031400 con 0x7f6ddc003f50 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.738+0000 7f6de7fff640 1 -- 172.21.15.146:0/604933306 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0009690 con 0x7f6ddc003f50 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df75c5640 1 -- 172.21.15.146:0/604933306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 msgr2=0x7f6ddc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df75c5640 1 --2- 172.21.15.146:0/604933306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6ddc004800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009ac0 tx=0x7f6ddc005830 comp rx=0 tx=0).stop 2024-04-25T15:21:47.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df5b62640 1 -- 172.21.15.146:0/604933306 reap_dead start 2024-04-25T15:21:47.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df75c5640 1 -- 172.21.15.146:0/604933306 shutdown_connections 2024-04-25T15:21:47.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df75c5640 1 -- 172.21.15.146:0/604933306 >> 172.21.15.146:0/604933306 conn(0x7f6df00b7460 msgr2=0x7f6df00b7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df75c5640 1 -- 172.21.15.146:0/604933306 shutdown_connections 2024-04-25T15:21:47.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.739+0000 7f6df75c5640 1 -- 172.21.15.146:0/604933306 wait complete. 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.741+0000 7f6df75c5640 1 Processor -- start 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.741+0000 7f6df75c5640 1 -- start start 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df75c5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6df014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df75c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df0152520 0x7f6df00c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df75c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df0154bc0 0x7f6df0154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df75c5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6df006ddf0 con 0x7f6ddc003f50 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df75c5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6df00cfa60 con 0x7f6df0154bc0 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df75c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6df0165cc0 con 0x7f6df0152520 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df5361640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6df014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df5361640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6df014fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40298/0 (socket says 172.21.15.146:40298) 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df5361640 1 -- 172.21.15.146:0/3787661739 learned_addr learned my addr 172.21.15.146:0/3787661739 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df5b62640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df0154bc0 0x7f6df0154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.742+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df0152520 0x7f6df00c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df5361640 1 -- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df0152520 msgr2=0x7f6df00c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df5361640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df0152520 0x7f6df00c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df5361640 1 -- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df0154bc0 msgr2=0x7f6df0154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df5361640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df0154bc0 0x7f6df0154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df5361640 1 -- 172.21.15.146:0/3787661739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df0153b20 con 0x7f6ddc003f50 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df0152520 0x7f6df00c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df5361640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6df014fde0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0072c0 tx=0x7f6dec007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6de5ffb640 1 -- 172.21.15.146:0/3787661739 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00f960 con 0x7f6ddc003f50 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6de5ffb640 1 -- 172.21.15.146:0/3787661739 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dec018070 con 0x7f6ddc003f50 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6de5ffb640 1 -- 172.21.15.146:0/3787661739 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00fd60 con 0x7f6ddc003f50 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df0153e00 con 0x7f6ddc003f50 2024-04-25T15:21:47.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.743+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df01543c0 con 0x7f6ddc003f50 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.745+0000 7f6de5ffb640 1 -- 172.21.15.146:0/3787661739 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f6dec01d460 con 0x7f6ddc003f50 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.745+0000 7f6de5ffb640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6dd4074570 0x7f6dd4076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.745+0000 7f6de5ffb640 1 -- 172.21.15.146:0/3787661739 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f6dec095520 con 0x7f6ddc003f50 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.745+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6dd4074570 0x7f6dd4076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.746+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6dd4074570 0x7f6dd4076a80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009ac0 tx=0x7f6de0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.746+0000 7f6dfe477740 1 Processor -- start 2024-04-25T15:21:47.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.746+0000 7f6dfe477740 1 -- start start 2024-04-25T15:21:47.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.747+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.747+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd85c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.747+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.747+0000 7f6dfe477740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5560dcca4170 con 0x5560dc974d00 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.747+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5560dcc3fb80 con 0x5560dc99dbc0 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.747+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5560dcca0130 con 0x5560dc97cee0 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd85c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd85c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40302/0 (socket says 172.21.15.146:40302) 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 -- 172.21.15.146:0/1131412887 learned_addr learned my addr 172.21.15.146:0/1131412887 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df5361640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df5b62640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 -- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 msgr2=0x5560dcd861a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd861a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 -- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 msgr2=0x5560dcd6af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 -- 172.21.15.146:0/1131412887 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560dcc61510 con 0x5560dc974d00 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df5b62640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:47.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df5361640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:47.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd85c60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6de002ab10 tx=0x7f6de002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5830b2fee9ad9bc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6dd2ffd640 1 -- 172.21.15.146:0/1131412887 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0040040 con 0x5560dc974d00 2024-04-25T15:21:47.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6dd2ffd640 1 -- 172.21.15.146:0/1131412887 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de0003d60 con 0x5560dc974d00 2024-04-25T15:21:47.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.748+0000 7f6dd2ffd640 1 -- 172.21.15.146:0/1131412887 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0036040 con 0x5560dc974d00 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 -- 172.21.15.146:0/1131412887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 msgr2=0x5560dcd85c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd85c60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6de002ab10 tx=0x7f6de002e000 comp rx=0 tx=0).stop 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 -- 172.21.15.146:0/1131412887 shutdown_connections 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd85c60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1131412887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 -- 172.21.15.146:0/1131412887 >> 172.21.15.146:0/1131412887 conn(0x5560dc991d10 msgr2=0x5560dcd40590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 -- 172.21.15.146:0/1131412887 shutdown_connections 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.749+0000 7f6dfe477740 1 -- 172.21.15.146:0/1131412887 wait complete. 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.750+0000 7f6dfe477740 1 -- start start 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.750+0000 7f6dfe477740 1 Processor -- start 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.751+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.751+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc97cee0 0x5560dcd7e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.751+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc99dbc0 0x5560dcd671a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.751+0000 7f6dfe477740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5560dcc2aa90 con 0x5560dc974d00 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.751+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5560dcd6c770 con 0x5560dc97cee0 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.751+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5560dcd66c10 con 0x5560dc99dbc0 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dfe477740 10 client.0 _ll_register_callbacks cb 0x5560dcc4aff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd6be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40312/0 (socket says 172.21.15.146:40312) 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 -- 172.21.15.146:0/3234189996 learned_addr learned my addr 172.21.15.146:0/3234189996 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5b62640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc99dbc0 0x5560dcd671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 -- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc99dbc0 msgr2=0x5560dcd671a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc97cee0 0x5560dcd7e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc99dbc0 0x5560dcd671a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 -- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc97cee0 msgr2=0x5560dcd7e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc97cee0 0x5560dcd7e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 -- 172.21.15.146:0/3234189996 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560dca82030 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6df5361640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd6be40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6dec05ba00 tx=0x7f6dec095420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9cb654949af81f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dcbfff640 1 -- 172.21.15.146:0/3234189996 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec091060 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dcbfff640 1 -- 172.21.15.146:0/3234189996 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f6dec01d750 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de0003d60 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dcbfff640 1 -- 172.21.15.146:0/3234189996 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f6dec07d050 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dfe477740 10 client.6012 fetch_fsmap learned FSMap version 17 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5560dc9b5c20 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.752+0000 7f6dcbfff640 1 -- 172.21.15.146:0/3234189996 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de0003d60 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dcbfff640 1 -- 172.21.15.146:0/3234189996 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f6dec0932b0 con 0x5560dc974d00 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dfe477740 10 client.6012 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dfe477740 10 client.6012 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.b 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dfe477740 20 client.6012 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:47.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dc95c790 0x5560dcd676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dfe477740 4 client.6012 mds_command: new command op to 5556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 --> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560dccb1d80 con 0x5560dc95c790 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dc95c790 0x5560dcd676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6dcbfff640 10 client.6012 ms_handle_connect on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.753+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dc95c790 0x5560dcd676e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51e67ace9d7885cc server_cookie=d710f2a7d71589a8 in_seq=0 out_seq=0 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dcbfff640 1 -- 172.21.15.146:0/3234189996 <== mds.0 v2:172.21.15.177:6833/539736758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5560dccb1d80 con 0x5560dc95c790 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dcbfff640 10 client.6012 handle_command_reply: tid=0 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 client.6012 shutdown 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 msgr2=0x5560dcd6be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd6be40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6dec05ba00 tx=0x7f6dec095420 comp rx=0 tx=0).stop 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 shutdown_connections 2024-04-25T15:21:47.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc99dbc0 0x5560dcd671a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dc95c790 0x5560dcd676e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc97cee0 0x5560dcd7e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3234189996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dcd6be40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dcbfff640 0 client.6012 ms_handle_reset on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.754+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 >> 172.21.15.146:0/3234189996 conn(0x5560dc991d10 msgr2=0x5560dcd6af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.755+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 shutdown_connections 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.755+0000 7f6dfe477740 1 -- 172.21.15.146:0/3234189996 wait complete. 2024-04-25T15:21:47.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.755+0000 7f6dfe477740 20 client.6012 trim_cache size 0 max 16384 2024-04-25T15:21:47.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.772+0000 7f6dfe477740 1 Processor -- start 2024-04-25T15:21:47.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.772+0000 7f6dfe477740 1 -- start start 2024-04-25T15:21:47.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc974d00 0x5560dcd67290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6dfe477740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0201540 con 0x5560dc99dbc0 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6df0200540 con 0x5560dc974d00 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6df01ff970 con 0x5560dc97cee0 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6df5b62640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6df5b62640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40326/0 (socket says 172.21.15.146:40326) 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.773+0000 7f6df5b62640 1 -- 172.21.15.146:0/3652262509 learned_addr learned my addr 172.21.15.146:0/3652262509 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5361640 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc974d00 0x5560dcd67290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df4b60640 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5b62640 1 -- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 msgr2=0x5560dcd6af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5b62640 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5b62640 1 -- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc974d00 msgr2=0x5560dcd67290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5b62640 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc974d00 0x5560dcd67290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5b62640 1 -- 172.21.15.146:0/3652262509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560dc975920 con 0x5560dc99dbc0 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5361640 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc974d00 0x5560dcd67290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6df5b62640 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6df00d25a0 tx=0x7f6de8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd14ed379d7c658 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6dd27fc640 1 -- 172.21.15.146:0/3652262509 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8018620 con 0x5560dc99dbc0 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6dd27fc640 1 -- 172.21.15.146:0/3652262509 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de80187c0 con 0x5560dc99dbc0 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.774+0000 7f6dd27fc640 1 -- 172.21.15.146:0/3652262509 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8018bb0 con 0x5560dc99dbc0 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 -- 172.21.15.146:0/3652262509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 msgr2=0x5560dcd6b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6df00d25a0 tx=0x7f6de8009750 comp rx=0 tx=0).stop 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 -- 172.21.15.146:0/3652262509 shutdown_connections 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 --2- 172.21.15.146:0/3652262509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc974d00 0x5560dcd67290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 -- 172.21.15.146:0/3652262509 >> 172.21.15.146:0/3652262509 conn(0x5560dc991d10 msgr2=0x5560dcd67de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 -- 172.21.15.146:0/3652262509 shutdown_connections 2024-04-25T15:21:47.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.775+0000 7f6dfe477740 1 -- 172.21.15.146:0/3652262509 wait complete. 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.776+0000 7f6dfe477740 1 -- start start 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.776+0000 7f6dfe477740 1 Processor -- start 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dce01cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6df01ffef0 con 0x5560dc974d00 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5560dcd3d250 con 0x5560dc99dbc0 2024-04-25T15:21:47.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5560dcd7cf00 con 0x5560dc97cee0 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6dfe477740 10 client.0 _ll_register_callbacks cb 0x7f6dec00ff00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dce01cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dce01cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40342/0 (socket says 172.21.15.146:40342) 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 -- 172.21.15.146:0/1924779837 learned_addr learned my addr 172.21.15.146:0/1924779837 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5b62640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 -- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 msgr2=0x5560dcd6af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 -- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 msgr2=0x5560dcd6b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.777+0000 7f6df5361640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6df5361640 1 -- 172.21.15.146:0/1924779837 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560dcd65990 con 0x5560dc974d00 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6df5361640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dce01cf0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6dec001970 tx=0x7f6dec062070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca030d24f49f1b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dcbfff640 1 -- 172.21.15.146:0/1924779837 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec078070 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dcbfff640 1 -- 172.21.15.146:0/1924779837 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f6dec01d750 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de80187c0 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dcbfff640 1 -- 172.21.15.146:0/1924779837 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f6dec05fb10 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 10 client.6018 fetch_fsmap learned FSMap version 17 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5560dcd866a0 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dcbfff640 1 -- 172.21.15.146:0/1924779837 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de80187c0 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dcbfff640 1 -- 172.21.15.146:0/1924779837 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f6dec05ad80 con 0x5560dc974d00 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 10 client.6018 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 10 client.6018 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.b 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 20 client.6018 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dcd6c900 0x5560dcd40280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 4 client.6018 mds_command: new command op to 5556 tid=0 multi_id=0 [{"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f", "wait": true}] 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.778+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 --> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] -- command(tid 0: {"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f", "wait": true}) -- 0x5560dccabbf0 con 0x5560dcd6c900 2024-04-25T15:21:47.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.779+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dcd6c900 0x5560dcd40280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.779+0000 7f6dcbfff640 10 client.6018 ms_handle_connect on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:47.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.779+0000 7f6df4b60640 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dcd6c900 0x5560dcd40280 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fd012f1ebd91b4d server_cookie=6c96434b47e08c4e in_seq=0 out_seq=0 2024-04-25T15:21:47.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.786+0000 7f6dcbfff640 1 -- 172.21.15.146:0/1924779837 <== mds.0 v2:172.21.15.177:6833/539736758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2332 (crc 0 0 0) 0x5560dccabbf0 con 0x5560dcd6c900 2024-04-25T15:21:47.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.786+0000 7f6dcbfff640 10 client.6018 handle_command_reply: tid=0 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.786+0000 7f6dfe477740 1 client.6018 shutdown 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.786+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 msgr2=0x5560dce01cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.786+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dce01cf0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6dec001970 tx=0x7f6dec062070 comp rx=0 tx=0).stop 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 shutdown_connections 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x5560dcd6c900 0x5560dcd40280 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560dc99dbc0 0x5560dcd6b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560dc97cee0 0x5560dcd6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 --2- 172.21.15.146:0/1924779837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5560dc974d00 0x5560dce01cf0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dcbfff640 0 client.6018 ms_handle_reset on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 >> 172.21.15.146:0/1924779837 conn(0x5560dc991d10 msgr2=0x5560dcd67290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 shutdown_connections 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 1 -- 172.21.15.146:0/1924779837 wait complete. 2024-04-25T15:21:47.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6dfe477740 20 client.6018 trim_cache size 0 max 16384 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "op": { 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "result": 0, 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "quiesce complete", 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "reqid": { 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": { 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "mds", 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 0 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-04-25T15:21:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "op_type": "internal_op", 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "internal_op": 5384, 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "op_name": "quiesce_path", 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.780962+0000", 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated" 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.780962+0000", 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read" 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.780962+0000", 2024-04-25T15:21:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled" 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.780962+0000", 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read" 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.780962+0000", 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched" 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.781031+0000", 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.781054+0000", 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.781073+0000", 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.781093+0000", 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.787613+0000", 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.787625+0000", 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.787635+0000", 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.787645+0000", 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:21:47.787661+0000", 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "quiesce complete" 2024-04-25T15:21:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "locks": null 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "state": { 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes": 2, 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_quiesced": 1, 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_blocked": 1, 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_dropped": 0, 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "failed": [] 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:21:47.795 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6dd4074570 msgr2=0x7f6dd4076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6dd4074570 0x7f6dd4076a80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009ac0 tx=0x7f6de0008030 comp rx=0 tx=0).stop 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 msgr2=0x7f6df014fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6df014fde0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0072c0 tx=0x7f6dec007780 comp rx=0 tx=0).stop 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 shutdown_connections 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6dd4074570 0x7f6dd4076a80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df0154bc0 0x7f6df0154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6df0152520 0x7f6df00c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 --2- 172.21.15.146:0/3787661739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ddc003f50 0x7f6df014fde0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 >> 172.21.15.146:0/3787661739 conn(0x7f6df00b7460 msgr2=0x7f6df015ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 shutdown_connections 2024-04-25T15:21:47.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.787+0000 7f6df75c5640 1 -- 172.21.15.146:0/3787661739 wait complete. 2024-04-25T15:21:47.800 DEBUG:tasks.cephfs.test_quiesce:{'op': {'result': 0, 'flag_point': 'quiesce complete', 'reqid': {'entity': {'type': 'mds', 'num': 0}, 'tid': 1}, 'op_type': 'internal_op', 'internal_op': 5384, 'op_name': 'quiesce_path', 'events': [{'time': '2024-04-25T15:21:47.780962+0000', 'event': 'initiated'}, {'time': '2024-04-25T15:21:47.780962+0000', 'event': 'header_read'}, {'time': '2024-04-25T15:21:47.780962+0000', 'event': 'throttled'}, {'time': '2024-04-25T15:21:47.780962+0000', 'event': 'all_read'}, {'time': '2024-04-25T15:21:47.780962+0000', 'event': 'dispatched'}, {'time': '2024-04-25T15:21:47.781031+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.781054+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.781073+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.781093+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.787613+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.787625+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.787635+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.787645+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:21:47.787661+0000', 'event': 'quiesce complete'}], 'locks': None}, 'state': {'inodes': 2, 'inodes_quiesced': 1, 'inodes_blocked': 1, 'inodes_dropped': 0, 'failed': []}} 2024-04-25T15:21:47.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:21:47.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.907+0000 7fa1e213f640 1 Processor -- start 2024-04-25T15:21:47.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.907+0000 7fa1e213f640 1 -- start start 2024-04-25T15:21:47.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1e213f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa1dc0b8dc0 0x7fa1dc071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1e213f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa1dc161ab0 0x7fa1dc161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1e213f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1dc0cfb60 con 0x7fa1dc0cf040 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1e213f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa1dc0d2a20 con 0x7fa1dc161ab0 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1e213f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa1dc0d2460 con 0x7fa1dc0b8dc0 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1dbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa1dc161ab0 0x7fa1dc161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1daffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa1dc0cf040 0x7fa1dc0cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:51190/0 (socket says 172.21.15.146:51190) 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.908+0000 7fa1daffd640 1 -- 172.21.15.146:0/3858345046 learned_addr learned my addr 172.21.15.146:0/3858345046 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1db7fe640 1 --2- 172.21.15.146:0/3858345046 >> v2:172.21.15.177:3300/0 conn(0x7fa1dc0b8dc0 0x7fa1dc071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1dbfff640 1 -- 172.21.15.146:0/3858345046 >> v2:172.21.15.177:3300/0 conn(0x7fa1dc0b8dc0 msgr2=0x7fa1dc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1dbfff640 1 --2- 172.21.15.146:0/3858345046 >> v2:172.21.15.177:3300/0 conn(0x7fa1dc0b8dc0 0x7fa1dc071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1dbfff640 1 -- 172.21.15.146:0/3858345046 >> v1:172.21.15.146:6789/0 conn(0x7fa1dc0cf040 legacy=0x7fa1dc0cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1dbfff640 1 -- 172.21.15.146:0/3858345046 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc0d2740 con 0x7fa1dc161ab0 2024-04-25T15:21:47.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1da7fc640 1 -- 172.21.15.146:0/3858345046 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 307317182 0 0) 0x7fa1dc0cfb60 con 0x7fa1dc0cf040 2024-04-25T15:21:47.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.909+0000 7fa1dbfff640 1 --2- 172.21.15.146:0/3858345046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc161ec0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc074870 tx=0x7fa1d402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396ef3f3e0e91ebe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1da7fc640 1 -- 172.21.15.146:0/3858345046 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4007740 con 0x7fa1dc161ab0 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1da7fc640 1 -- 172.21.15.146:0/3858345046 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d40078e0 con 0x7fa1dc161ab0 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1da7fc640 1 -- 172.21.15.146:0/3858345046 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4007cb0 con 0x7fa1dc161ab0 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 -- 172.21.15.146:0/3858345046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 msgr2=0x7fa1dc161ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 --2- 172.21.15.146:0/3858345046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc161ec0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc074870 tx=0x7fa1d402a5f0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 -- 172.21.15.146:0/3858345046 shutdown_connections 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 --2- 172.21.15.146:0/3858345046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc161ec0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 --2- 172.21.15.146:0/3858345046 >> v2:172.21.15.177:3300/0 conn(0x7fa1dc0b8dc0 0x7fa1dc071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 -- 172.21.15.146:0/3858345046 >> 172.21.15.146:0/3858345046 conn(0x7fa1dc0b5360 msgr2=0x7fa1dc0b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 -- 172.21.15.146:0/3858345046 shutdown_connections 2024-04-25T15:21:47.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.910+0000 7fa1e213f640 1 -- 172.21.15.146:0/3858345046 wait complete. 2024-04-25T15:21:47.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.912+0000 7fa1e213f640 1 Processor -- start 2024-04-25T15:21:47.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.912+0000 7fa1e213f640 1 -- start start 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1dc0b8dc0 0x7fa1dc15fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1dc0cf040 0x7fa1dc1f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc1f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc071b60 con 0x7fa1dc161ab0 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc0ce4f0 con 0x7fa1dc0b8dc0 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa1dc0d20f0 con 0x7fa1dc0cf040 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc1f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc1f35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40356/0 (socket says 172.21.15.146:40356) 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 -- 172.21.15.146:0/319632359 learned_addr learned my addr 172.21.15.146:0/319632359 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1db7fe640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1dc0b8dc0 0x7fa1dc15fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1daffd640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1dc0cf040 0x7fa1dc1f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 -- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1dc0cf040 msgr2=0x7fa1dc1f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1dc0cf040 0x7fa1dc1f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 -- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1dc0b8dc0 msgr2=0x7fa1dc15fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1dc0b8dc0 0x7fa1dc15fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 -- 172.21.15.146:0/319632359 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc1e6c70 con 0x7fa1dc161ab0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1daffd640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1dc0cf040 0x7fa1dc1f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1dbfff640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc1f35e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa1d402abe0 tx=0x7fa1d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d403f070 con 0x7fa1dc161ab0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d4031c90 con 0x7fa1dc161ab0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4044680 con 0x7fa1dc161ab0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1dc1e6f50 con 0x7fa1dc161ab0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.914+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1dc1e7510 con 0x7fa1dc161ab0 2024-04-25T15:21:47.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.915+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1dc06a0e0 con 0x7fa1dc161ab0 2024-04-25T15:21:47.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.916+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa1d403b410 con 0x7fa1dc161ab0 2024-04-25T15:21:47.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.916+0000 7fa1c3fff640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1c8074540 0x7fa1c8076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:47.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.916+0000 7fa1db7fe640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1c8074540 0x7fa1c8076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:47.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.917+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa1d40b6f30 con 0x7fa1dc161ab0 2024-04-25T15:21:47.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.917+0000 7fa1db7fe640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1c8074540 0x7fa1c8076a50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0d3020 tx=0x7fa1cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:47.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:47.921+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa1d4080f10 con 0x7fa1dc161ab0 2024-04-25T15:21:48.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.138+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1dc0b5f50 con 0x7fa1dc161ab0 2024-04-25T15:21:48.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.138+0000 7fa1c3fff640 1 -- 172.21.15.146:0/319632359 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5187 (secure 0 0 0) 0x7fa1d4085210 con 0x7fa1dc161ab0 2024-04-25T15:21:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:48.140 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:10.651740+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5556":{"gid":5556,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.177:6835/539736758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":539736758},{"type":"v1","addr":"172.21.15.177:6835","nonce":539736758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5556,"qdb_cluster":[5556]},"id":2}]} 2024-04-25T15:21:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 17 2024-04-25T15:21:48.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.140+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1c8074540 msgr2=0x7fa1c8076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.140+0000 7fa1e213f640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1c8074540 0x7fa1c8076a50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0d3020 tx=0x7fa1cc008040 comp rx=0 tx=0).stop 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.140+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 msgr2=0x7fa1dc1f35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.140+0000 7fa1e213f640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc1f35e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa1d402abe0 tx=0x7fa1d4004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 shutdown_connections 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa1c8074540 0x7fa1c8076a50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1dc161ab0 0x7fa1dc1f35e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1dc0cf040 0x7fa1dc1f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 --2- 172.21.15.146:0/319632359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1dc0b8dc0 0x7fa1dc15fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 >> 172.21.15.146:0/319632359 conn(0x7fa1dc0b5360 msgr2=0x7fa1dc071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 shutdown_connections 2024-04-25T15:21:48.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.141+0000 7fa1e213f640 1 -- 172.21.15.146:0/319632359 wait complete. 2024-04-25T15:21:48.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f 0 2024-04-25T15:21:48.283 INFO:teuthology.orchestra.run.smithi177.stdout:[{"ino":1099511627779,"rdev":0,"ctime":"2024-04-25T15:21:47.544331+0000","btime":"2024-04-25T15:21:22.441084+0000","mode":16895,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":6,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-04-25T15:21:47.544331+0000","atime":"2024-04-25T15:21:22.441084+0000","time_warp_seq":0,"change_attr":2,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-04-25T15:21:47.544331+0000","num_files":1,"num_subdirs":0,"change_attr":1},"rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-25T15:21:47.576833+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-25T15:21:47.576833+0000"},"version":20,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":2,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"waiter":0,"authpin":1},"nref":8,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{"gather_set":[],"state":"sync","type":"ipolicy","is_leased":false,"num_rdlocks":1,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{"reqid":{"entity":{"type":"mds","num":0},"tid":2}}},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":5189,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5204,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5799,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":7},{"client_id":5967,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":3},{"client_id":5985,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f/file","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511629282,"remote_type":"","version":6,"projected_version":6,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-04-25T15:21:48.285 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-04-25T15:21:47.576833+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-04-25T15:21:22.441084+0000", "auth_pins": 1, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-04-25T15:21:22.441084+0000", "change_attr": 2, "client_caps": [ { "client_id": 5189, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5204, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5799, "issued": "pAsLsXs", "last_sent": 7, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5967, "issued": "pAsLsXs", "last_sent": 3, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5985, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-04-25T15:21:47.544331+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511629282, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f/file", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 6, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 6, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 1, "mtime": "2024-04-25T15:21:47.544331+0000", "num_files": 1, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1099511627779, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 6, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16895, "mtime": "2024-04-25T15:21:47.544331+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 8, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 2, "request": 1, "waiter": 0 }, "policylock": { "gather_set": [], "is_leased": false, "num_rdlocks": 1, "num_wrlocks": 0, "num_xlocks": 0, "state": "sync", "type": "ipolicy", "xlock_by": {} }, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "lock", "type": "iquiesce", "xlock_by": { "reqid": { "entity": { "num": 0, "type": "mds" }, "tid": 2 } } }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-04-25T15:21:47.576833+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 20, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-04-25T15:21:48.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 ops --flags=locks --path=/tmp/mds.0-ops 2024-04-25T15:21:48.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.395+0000 7f0611a67640 1 Processor -- start 2024-04-25T15:21:48.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.395+0000 7f0611a67640 1 -- start start 2024-04-25T15:21:48.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f0611a67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 0x7f060c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f0611a67640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f060c06c550 0x7f060c06c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f0611a67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f060c0b2fe0 con 0x7f060c0d24d0 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f0611a67640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f060c0b3820 con 0x7f060c06c550 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f0611a67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f060c0b2850 con 0x7f060c0b7430 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f060c0d24d0 0x7f060c0d28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:51204/0 (socket says 172.21.15.146:51204) 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060bfff640 1 -- 172.21.15.146:0/2689262404 learned_addr learned my addr 172.21.15.146:0/2689262404 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060affd640 1 --2- 172.21.15.146:0/2689262404 >> v2:172.21.15.177:3300/0 conn(0x7f060c06c550 0x7f060c06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625907856 0 0) 0x7f060c0b2fe0 con 0x7f060c0d24d0 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060b7fe640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 0x7f060c0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05f4003520 con 0x7f060c0d24d0 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3589417164 0 0) 0x7f05f4003520 con 0x7f060c0d24d0 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 msgr2=0x7f060c0b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 0x7f060c0b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 >> v2:172.21.15.177:3300/0 conn(0x7f060c06c550 msgr2=0x7f060c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.397+0000 7f060a7fc640 1 --2- 172.21.15.146:0/2689262404 >> v2:172.21.15.177:3300/0 conn(0x7f060c06c550 0x7f060c06c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060c0d1ff0 con 0x7f060c0d24d0 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f06040019f0 con 0x7f060c0d24d0 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 >> v1:172.21.15.146:6789/0 conn(0x7f060c0d24d0 legacy=0x7f060c0d28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 0x7f05f40039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f4003f50 0x7f05f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05f4004d40 0x7f05f4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f060c0b3820 con 0x7f05f4004d40 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f060c0b2850 con 0x7f05f4003f50 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f05f400baf0 con 0x7f060c0b7430 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060bfff640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05f4004d40 0x7f05f4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060b7fe640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 0x7f05f40039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060affd640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f4003f50 0x7f05f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060bfff640 1 -- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 msgr2=0x7f05f40039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060bfff640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c0b7430 0x7f05f40039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060bfff640 1 -- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f4003f50 msgr2=0x7f05f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060bfff640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05f4003f50 0x7f05f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.398+0000 7f060bfff640 1 -- 172.21.15.146:0/2689262404 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f4003520 con 0x7f05f4004d40 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f060bfff640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05f4004d40 0x7f05f4005150 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f060400a870 tx=0x7f05f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4caca34a82a4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060400a4c0 con 0x7f05f4004d40 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060402f070 con 0x7f05f4004d40 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f060a7fc640 1 -- 172.21.15.146:0/2689262404 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060402bcc0 con 0x7f05f4004d40 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f0611a67640 1 -- 172.21.15.146:0/2689262404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05f4004d40 msgr2=0x7f05f4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f0611a67640 1 --2- 172.21.15.146:0/2689262404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05f4004d40 0x7f05f4005150 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f060400a870 tx=0x7f05f4005830 comp rx=0 tx=0).stop 2024-04-25T15:21:48.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.399+0000 7f060bfff640 1 -- 172.21.15.146:0/2689262404 reap_dead start 2024-04-25T15:21:48.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.400+0000 7f0611a67640 1 -- 172.21.15.146:0/2689262404 shutdown_connections 2024-04-25T15:21:48.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.400+0000 7f0611a67640 1 -- 172.21.15.146:0/2689262404 >> 172.21.15.146:0/2689262404 conn(0x7f060c09b2b0 msgr2=0x7f060c0b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.400+0000 7f0611a67640 1 -- 172.21.15.146:0/2689262404 shutdown_connections 2024-04-25T15:21:48.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.400+0000 7f0611a67640 1 -- 172.21.15.146:0/2689262404 wait complete. 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.401+0000 7f0611a67640 1 Processor -- start 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.402+0000 7f0611a67640 1 -- start start 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f0611a67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c1e52e0 0x7f060c09f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f0611a67640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 0x7f060c1f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f0611a67640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f060c15d420 0x7f060c1f7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f0611a67640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f060c160280 con 0x7f060c15d010 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f0611a67640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f060c0b3540 con 0x7f060c15d420 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f0611a67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f060c160400 con 0x7f060c1e52e0 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 0x7f060c1f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 0x7f060c1f7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40362/0 (socket says 172.21.15.146:40362) 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 -- 172.21.15.146:0/3571125947 learned_addr learned my addr 172.21.15.146:0/3571125947 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060bfff640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f060c15d420 0x7f060c1f7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060b7fe640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c1e52e0 0x7f060c09f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 -- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c1e52e0 msgr2=0x7f060c09f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c1e52e0 0x7f060c09f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 -- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f060c15d420 msgr2=0x7f060c1f7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f060c15d420 0x7f060c1f7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 -- 172.21.15.146:0/3571125947 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060c1e57e0 con 0x7f060c15d010 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.403+0000 7f060affd640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 0x7f060c1f7710 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0600011960 tx=0x7f0600011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.404+0000 7f05f3fff640 1 -- 172.21.15.146:0/3571125947 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060000fa30 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.404+0000 7f05f3fff640 1 -- 172.21.15.146:0/3571125947 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060000fbd0 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.404+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f060c1e5ac0 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.404+0000 7f05f3fff640 1 -- 172.21.15.146:0/3571125947 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600014440 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.404+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f060c1e6080 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.406+0000 7f05f3fff640 1 -- 172.21.15.146:0/3571125947 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f0600012040 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.406+0000 7f05f3fff640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f05f8074570 0x7f05f8076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.406+0000 7f05f3fff640 1 -- 172.21.15.146:0/3571125947 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0600095930 con 0x7f060c15d010 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.406+0000 7f060b7fe640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f05f8074570 0x7f05f8076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.406+0000 7f060b7fe640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f05f8074570 0x7f05f8076a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f05fc009bc0 tx=0x7f05fc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.407+0000 7f06189fb740 1 Processor -- start 2024-04-25T15:21:48.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.407+0000 7f06189fb740 1 -- start start 2024-04-25T15:21:48.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f06189fb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 0x56113b0eae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ace36e0 0x56113b0eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f06189fb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56113b00a690 con 0x56113acdb500 2024-04-25T15:21:48.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56113afc45d0 con 0x56113ace36e0 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x56113b006650 con 0x56113ad043c0 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 0x56113b0eae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 0x56113b0eae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40378/0 (socket says 172.21.15.146:40378) 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f060b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f060affd640 1 -- 172.21.15.146:0/3686430305 learned_addr learned my addr 172.21.15.146:0/3686430305 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.408+0000 7f060bfff640 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ace36e0 0x56113b0eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f060affd640 1 -- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 msgr2=0x56113b0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f060affd640 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f060affd640 1 -- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ace36e0 msgr2=0x56113b0eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f060affd640 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ace36e0 0x56113b0eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f060affd640 1 -- 172.21.15.146:0/3686430305 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56113afd1110 con 0x56113acdb500 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f060affd640 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 0x56113b0eae90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0600012010 tx=0x7f060005f400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac9d90468675c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f05f17fa640 1 -- 172.21.15.146:0/3686430305 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600022070 con 0x56113acdb500 2024-04-25T15:21:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f05f17fa640 1 -- 172.21.15.146:0/3686430305 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0600063be0 con 0x56113acdb500 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.409+0000 7f05f17fa640 1 -- 172.21.15.146:0/3686430305 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060005fcd0 con 0x56113acdb500 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 -- 172.21.15.146:0/3686430305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 msgr2=0x56113b0eae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 0x56113b0eae90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0600012010 tx=0x7f060005f400 comp rx=0 tx=0).stop 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 -- 172.21.15.146:0/3686430305 shutdown_connections 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ace36e0 0x56113b0eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113acdb500 0x56113b0eae90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 --2- 172.21.15.146:0/3686430305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 -- 172.21.15.146:0/3686430305 >> 172.21.15.146:0/3686430305 conn(0x56113acf8510 msgr2=0x56113b0a57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 -- 172.21.15.146:0/3686430305 shutdown_connections 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.410+0000 7f06189fb740 1 -- 172.21.15.146:0/3686430305 wait complete. 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.411+0000 7f06189fb740 1 -- start start 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.411+0000 7f06189fb740 1 Processor -- start 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113acdb500 0x56113b0f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0fa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ad043c0 0x56113b09f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56113af91cd0 con 0x56113ace36e0 2024-04-25T15:21:48.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56113b0cb010 con 0x56113ad043c0 2024-04-25T15:21:48.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x56113b0fa9b0 con 0x56113acdb500 2024-04-25T15:21:48.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f06189fb740 10 client.0 _ll_register_callbacks cb 0x56113afc2220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:48.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0fa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0fa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40386/0 (socket says 172.21.15.146:40386) 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060affd640 1 -- 172.21.15.146:0/1820741794 learned_addr learned my addr 172.21.15.146:0/1820741794 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060b7fe640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113acdb500 0x56113b0f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060bfff640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ad043c0 0x56113b09f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060affd640 1 -- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113acdb500 msgr2=0x56113b0f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.412+0000 7f060affd640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113acdb500 0x56113b0f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f060affd640 1 -- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ad043c0 msgr2=0x56113b09f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f060affd640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ad043c0 0x56113b09f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f060affd640 1 -- 172.21.15.146:0/1820741794 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56113ad1c420 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f060affd640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0fa3a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f06000662b0 tx=0x7f0600094000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed083ed13c8ae0af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f05ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f05ea7fc640 1 -- 172.21.15.146:0/1820741794 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060005fd20 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f05ea7fc640 1 -- 172.21.15.146:0/1820741794 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0600060b60 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0600063be0 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f05ea7fc640 1 -- 172.21.15.146:0/1820741794 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f0600068600 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 10 client.6039 fetch_fsmap learned FSMap version 17 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x56113ade8830 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f05ea7fc640 1 -- 172.21.15.146:0/1820741794 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0600063be0 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f05ea7fc640 1 -- 172.21.15.146:0/1820741794 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f060005ea90 con 0x56113ace36e0 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 10 client.6039 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 10 client.6039 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.b 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 20 client.6039 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113acc31a0 0x56113b0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 4 client.6039 mds_command: new command op to 5556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 --> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56113b0182a0 con 0x56113acc31a0 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.413+0000 7f060b7fe640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113acc31a0 0x56113b0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.414+0000 7f060b7fe640 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113acc31a0 0x56113b0ca080 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a83441ab5cc9230 server_cookie=ff0a229ae8a7298c in_seq=0 out_seq=0 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.414+0000 7f05ea7fc640 10 client.6039 ms_handle_connect on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.414+0000 7f05ea7fc640 1 -- 172.21.15.146:0/1820741794 <== mds.0 v2:172.21.15.177:6833/539736758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56113b0182a0 con 0x56113acc31a0 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.414+0000 7f05ea7fc640 10 client.6039 handle_command_reply: tid=0 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.414+0000 7f06189fb740 1 client.6039 shutdown 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 msgr2=0x56113b0fa3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0fa3a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f06000662b0 tx=0x7f0600094000 comp rx=0 tx=0).stop 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 shutdown_connections 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113acc31a0 0x56113b0ca080 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113ad043c0 0x56113b09f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0fa3a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f05ea7fc640 0 client.6039 ms_handle_reset on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:48.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 --2- 172.21.15.146:0/1820741794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113acdb500 0x56113b0f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 >> 172.21.15.146:0/1820741794 conn(0x56113acf8510 msgr2=0x56113b0d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 shutdown_connections 2024-04-25T15:21:48.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 1 -- 172.21.15.146:0/1820741794 wait complete. 2024-04-25T15:21:48.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.415+0000 7f06189fb740 20 client.6039 trim_cache size 0 max 16384 2024-04-25T15:21:48.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.431+0000 7f06189fb740 1 Processor -- start 2024-04-25T15:21:48.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.431+0000 7f06189fb740 1 -- start start 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.432+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b0f6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.432+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ace36e0 0x56113b0a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.432+0000 7f06189fb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 0x56113b0a4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.432+0000 7f06189fb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f060c203cd0 con 0x56113ad043c0 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.432+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f060c202cf0 con 0x56113acdb500 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.432+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f060c202050 con 0x56113ace36e0 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 0x56113b0a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 0x56113b0a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40392/0 (socket says 172.21.15.146:40392) 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 -- 172.21.15.146:0/1288695405 learned_addr learned my addr 172.21.15.146:0/1288695405 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:48.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060affd640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ace36e0 0x56113b0a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060b7fe640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b0f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 -- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ace36e0 msgr2=0x56113b0a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ace36e0 0x56113b0a3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 -- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 msgr2=0x56113b0f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b0f6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 -- 172.21.15.146:0/1288695405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56113b0a7b40 con 0x56113ad043c0 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060affd640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ace36e0 0x56113b0a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060b7fe640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b0f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f060bfff640 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 0x56113b0a4020 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f060402b530 tx=0x7f060402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a23f663757b8652 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f05f0ff9640 1 -- 172.21.15.146:0/1288695405 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604004b60 con 0x56113ad043c0 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f05f0ff9640 1 -- 172.21.15.146:0/1288695405 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060402f070 con 0x56113ad043c0 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.433+0000 7f05f0ff9640 1 -- 172.21.15.146:0/1288695405 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060402b880 con 0x56113ad043c0 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 -- 172.21.15.146:0/1288695405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 msgr2=0x56113b0a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 0x56113b0a4020 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f060402b530 tx=0x7f060402d000 comp rx=0 tx=0).stop 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 -- 172.21.15.146:0/1288695405 shutdown_connections 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ad043c0 0x56113b0a4020 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ace36e0 0x56113b0a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 --2- 172.21.15.146:0/1288695405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b0f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 -- 172.21.15.146:0/1288695405 >> 172.21.15.146:0/1288695405 conn(0x56113acf8510 msgr2=0x56113b0a6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 -- 172.21.15.146:0/1288695405 shutdown_connections 2024-04-25T15:21:48.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.434+0000 7f06189fb740 1 -- 172.21.15.146:0/1288695405 wait complete. 2024-04-25T15:21:48.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.435+0000 7f06189fb740 1 -- start start 2024-04-25T15:21:48.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.435+0000 7f06189fb740 1 Processor -- start 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b1676f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b16b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56113b0a7430 con 0x56113ace36e0 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f060c2026a0 con 0x56113acdb500 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x56113b0d18c0 con 0x56113ad043c0 2024-04-25T15:21:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f06189fb740 10 client.0 _ll_register_callbacks cb 0x7f0600063070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f060affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0d0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40400/0 (socket says 172.21.15.146:40400) 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.436+0000 7f060affd640 1 -- 172.21.15.146:0/2566400133 learned_addr learned my addr 172.21.15.146:0/2566400133 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060b7fe640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b1676f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060bfff640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b16b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060affd640 1 -- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 msgr2=0x56113b16b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060affd640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b16b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060affd640 1 -- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 msgr2=0x56113b1676f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060affd640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b1676f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060affd640 1 -- 172.21.15.146:0/2566400133 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56113b0e2390 con 0x56113ace36e0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060b7fe640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b1676f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f060affd640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0d0eb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f06000096b0 tx=0x7f060005bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e13fd9e0c95c23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f05ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f05ea7fc640 1 -- 172.21.15.146:0/2566400133 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600092060 con 0x56113ace36e0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f05ea7fc640 1 -- 172.21.15.146:0/2566400133 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f0600060d60 con 0x56113ace36e0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f060402f070 con 0x56113ace36e0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f05ea7fc640 1 -- 172.21.15.146:0/2566400133 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f060008ce60 con 0x56113ace36e0 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 10 client.6045 fetch_fsmap learned FSMap version 17 2024-04-25T15:21:48.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x56113b0d1600 con 0x56113ace36e0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f05ea7fc640 1 -- 172.21.15.146:0/2566400133 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060402f070 con 0x56113ace36e0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f05ea7fc640 1 -- 172.21.15.146:0/2566400133 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f060005e500 con 0x56113ace36e0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 10 client.6045 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 10 client.6045 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.b 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 20 client.6045 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113b0d06b0 0x56113b102e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 4 client.6045 mds_command: new command op to 5556 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}] 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.437+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 --> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}) -- 0x56113b0eb9d0 con 0x56113b0d06b0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.438+0000 7f060b7fe640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113b0d06b0 0x56113b102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.438+0000 7f05ea7fc640 10 client.6045 ms_handle_connect on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.438+0000 7f060b7fe640 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113b0d06b0 0x56113b102e40 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3b4c182f148be64 server_cookie=304c42e87c49058c in_seq=0 out_seq=0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f05ea7fc640 1 -- 172.21.15.146:0/2566400133 <== mds.0 v2:172.21.15.177:6833/539736758 1 ==== command_reply(tid 0: 0 ) ==== 8+0+33 (crc 0 0 0) 0x56113b0eb9d0 con 0x56113b0d06b0 2024-04-25T15:21:48.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f05ea7fc640 10 client.6045 handle_command_reply: tid=0 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 client.6045 shutdown 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 msgr2=0x56113b0d0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0d0eb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f06000096b0 tx=0x7f060005bfb0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 shutdown_connections 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56113ad043c0 0x56113b16b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56113ace36e0 0x56113b0d0eb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:6833/539736758,v1:172.21.15.177:6835/539736758] conn(0x56113b0d06b0 0x56113b102e40 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 --2- 172.21.15.146:0/2566400133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56113acdb500 0x56113b1676f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f05ea7fc640 0 client.6045 ms_handle_reset on v2:172.21.15.177:6833/539736758 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 >> 172.21.15.146:0/2566400133 conn(0x56113acf8510 msgr2=0x56113b0a6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 shutdown_connections 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 1 -- 172.21.15.146:0/2566400133 wait complete. 2024-04-25T15:21:48.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f06189fb740 20 client.6045 trim_cache size 0 max 16384 2024-04-25T15:21:48.449 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:48.449 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/tmp/mds.0-ops" 2024-04-25T15:21:48.449 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f05f8074570 msgr2=0x7f05f8076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f0611a67640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f05f8074570 0x7f05f8076a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f05fc009bc0 tx=0x7f05fc009190 comp rx=0 tx=0).stop 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 msgr2=0x7f060c1f7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.439+0000 7f0611a67640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 0x7f060c1f7710 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0600011960 tx=0x7f0600011d60 comp rx=0 tx=0).stop 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 shutdown_connections 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f05f8074570 0x7f05f8076a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f060c15d420 0x7f060c1f7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060c15d010 0x7f060c1f7710 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 --2- 172.21.15.146:0/3571125947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060c1e52e0 0x7f060c09f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:48.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 >> 172.21.15.146:0/3571125947 conn(0x7f060c09b2b0 msgr2=0x7f060c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:48.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 shutdown_connections 2024-04-25T15:21:48.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:48.440+0000 7f0611a67640 1 -- 172.21.15.146:0/3571125947 wait complete. 2024-04-25T15:21:48.452 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-25T15:21:48.467 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.brCrta57Ks 2024-04-25T15:21:48.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /tmp/mds.0-ops /tmp/tmp.brCrta57Ks 2024-04-25T15:21:48.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.brCrta57Ks 2024-04-25T15:21:48.655 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.brCrta57Ks is 8KB 2024-04-25T15:21:48.704 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.brCrta57Ks 2024-04-25T15:21:48.718 DEBUG:tasks.cephfs.filesystem:read 7762B of ops 2024-04-25T15:21:48.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/689b73eb-a2ee-47df-8f30-12bb7b0dea3f --path=/tmp/mds.0-cache 2024-04-25T15:21:48.881 INFO:teuthology.orchestra.run.smithi177.stdout:{"path":"/tmp/mds.0-cache"} 2024-04-25T15:21:48.882 DEBUG:tasks.cephfs.filesystem:_json_asok output { "path": "/tmp/mds.0-cache" } 2024-04-25T15:21:48.882 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-25T15:21:48.897 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.IWCXSmlurj 2024-04-25T15:21:48.897 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /tmp/mds.0-cache /tmp/tmp.IWCXSmlurj 2024-04-25T15:21:48.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.IWCXSmlurj 2024-04-25T15:21:49.081 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.IWCXSmlurj is 6KB 2024-04-25T15:21:49.129 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.IWCXSmlurj 2024-04-25T15:21:49.144 DEBUG:tasks.cephfs.filesystem:read 6486B of cache 2024-04-25T15:21:49.145 INFO:tasks.cephfs.test_quiesce:qp = 1; qi = 1; qib = 1; qina = 0 2024-04-25T15:21:49.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:21:49.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.257+0000 7f680df84640 1 Processor -- start 2024-04-25T15:21:49.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.257+0000 7f680df84640 1 -- start start 2024-04-25T15:21:49.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f680df84640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f68080cfbe0 0x7f68080cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f680df84640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68080d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f680df84640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68080b3920 0x7f68080b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f680df84640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f68080b5020 con 0x7f68080b3920 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f680df84640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f68080b4a60 con 0x7f68080cfbe0 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f680df84640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f68080b4d40 con 0x7f68080d19d0 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68080b3920 0x7f68080b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68080b3920 0x7f68080b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40414/0 (socket says 172.21.15.146:40414) 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 -- 172.21.15.146:0/911581208 learned_addr learned my addr 172.21.15.146:0/911581208 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6806ffd640 1 --2- 172.21.15.146:0/911581208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68080d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f68077fe640 1 --2- 172.21.15.146:0/911581208 >> v2:172.21.15.177:3300/0 conn(0x7f68080cfbe0 0x7f68080cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 -- 172.21.15.146:0/911581208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 msgr2=0x7f68080d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 --2- 172.21.15.146:0/911581208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68080d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 -- 172.21.15.146:0/911581208 >> v2:172.21.15.177:3300/0 conn(0x7f68080cfbe0 msgr2=0x7f68080cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 --2- 172.21.15.146:0/911581208 >> v2:172.21.15.177:3300/0 conn(0x7f68080cfbe0 0x7f68080cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 -- 172.21.15.146:0/911581208 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68080cead0 con 0x7f68080b3920 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f6807fff640 1 --2- 172.21.15.146:0/911581208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080b3920 0x7f68080b3d30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6808075260 tx=0x7f680002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0f763a39f88d4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f68067fc640 1 -- 172.21.15.146:0/911581208 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6800036070 con 0x7f68080b3920 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f68067fc640 1 -- 172.21.15.146:0/911581208 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6800031600 con 0x7f68080b3920 2024-04-25T15:21:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.260+0000 7f68067fc640 1 -- 172.21.15.146:0/911581208 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68000319d0 con 0x7f68080b3920 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 -- 172.21.15.146:0/911581208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080b3920 msgr2=0x7f68080b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 --2- 172.21.15.146:0/911581208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080b3920 0x7f68080b3d30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6808075260 tx=0x7f680002a730 comp rx=0 tx=0).stop 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 -- 172.21.15.146:0/911581208 shutdown_connections 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 --2- 172.21.15.146:0/911581208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080b3920 0x7f68080b3d30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 --2- 172.21.15.146:0/911581208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68080d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 --2- 172.21.15.146:0/911581208 >> v2:172.21.15.177:3300/0 conn(0x7f68080cfbe0 0x7f68080cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 -- 172.21.15.146:0/911581208 >> 172.21.15.146:0/911581208 conn(0x7f68080b73b0 msgr2=0x7f68080b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 -- 172.21.15.146:0/911581208 shutdown_connections 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.261+0000 7f680df84640 1 -- 172.21.15.146:0/911581208 wait complete. 2024-04-25T15:21:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.262+0000 7f680df84640 1 Processor -- start 2024-04-25T15:21:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.263+0000 7f680df84640 1 -- start start 2024-04-25T15:21:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f680df84640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68080b3920 0x7f68080af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f680df84640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 0x7f68080c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f680df84640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68081e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f680df84640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f68080cf7a0 con 0x7f68080cfbe0 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f680df84640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f68080d2320 con 0x7f68080b3920 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f680df84640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f68080afe00 con 0x7f68080d19d0 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 0x7f68080c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 0x7f68080c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40428/0 (socket says 172.21.15.146:40428) 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 -- 172.21.15.146:0/1860938378 learned_addr learned my addr 172.21.15.146:0/1860938378 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6807fff640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68081e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f68077fe640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68080b3920 0x7f68080af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 -- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 msgr2=0x7f68081e95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68081e95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 -- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68080b3920 msgr2=0x7f68080af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68080b3920 0x7f68080af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f6806ffd640 1 -- 172.21.15.146:0/1860938378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680806eac0 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.264+0000 7f68077fe640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68080b3920 0x7f68080af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.265+0000 7f6806ffd640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 0x7f68080c9a10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e870 tx=0x7f67fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.265+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc017070 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.265+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67fc00c940 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.265+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68081e8540 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.265+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00cd10 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.265+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68081e8ac0 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.266+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f680806a0e0 con 0x7f68080cfbe0 2024-04-25T15:21:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.266+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f67fc01a070 con 0x7f68080cfbe0 2024-04-25T15:21:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.267+0000 7f67effff640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f67f4074540 0x7f67f4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.267+0000 7f68077fe640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f67f4074540 0x7f67f4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.267+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f67fc0951d0 con 0x7f68080cfbe0 2024-04-25T15:21:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.268+0000 7f68077fe640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f67f4074540 0x7f67f4076a50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009a60 tx=0x7f67f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.273+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68081e8540 con 0x7f68080cfbe0 2024-04-25T15:21:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.484+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f68080cf920 con 0x7f68080cfbe0 2024-04-25T15:21:49.486 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-04-25T15:21:49.485+0000 7f79638be640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:21:49.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.486+0000 7f67effff640 1 -- 172.21.15.146:0/1860938378 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7f67fc05f170 con 0x7f68080cfbe0 2024-04-25T15:21:49.488 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:21:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f67f4074540 msgr2=0x7f67f4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f67f4074540 0x7f67f4076a50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009a60 tx=0x7f67f8008040 comp rx=0 tx=0).stop 2024-04-25T15:21:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 msgr2=0x7f68080c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 0x7f68080c9a10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e870 tx=0x7f67fc00ed30 comp rx=0 tx=0).stop 2024-04-25T15:21:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 shutdown_connections 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f67f4074540 0x7f67f4076a50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080d19d0 0x7f68081e95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68080cfbe0 0x7f68080c9a10 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 --2- 172.21.15.146:0/1860938378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68080b3920 0x7f68080af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 >> 172.21.15.146:0/1860938378 conn(0x7f68080b73b0 msgr2=0x7f6808070520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 shutdown_connections 2024-04-25T15:21:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.488+0000 7f680df84640 1 -- 172.21.15.146:0/1860938378 wait complete. 2024-04-25T15:21:49.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-25T15:21:49.550 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-04-25T15:21:49.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.612+0000 7fe0fc3ee640 1 Processor -- start 2024-04-25T15:21:49.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.612+0000 7fe0fc3ee640 1 -- start start 2024-04-25T15:21:49.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fc3ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fc3ee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 0x7fe0f40d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fc3ee640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0f40d2fd0 con 0x7fe0f409bd80 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fc3ee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe0f40ce920 con 0x7fe0f40d2680 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fc3ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe0f40cec00 con 0x7fe0f415e070 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fa98b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 0x7fe0f40d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fa18a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fa98b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 0x7fe0f40d2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40436/0 (socket says 172.21.15.146:40436) 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0f9989640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe0f409bd80 0x7fe0f40ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:35908/0 (socket says 172.21.15.146:35908) 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.613+0000 7fe0fa98b640 1 -- 172.21.15.146:0/206514896 learned_addr learned my addr 172.21.15.146:0/206514896 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0fa18a640 1 -- 172.21.15.146:0/206514896 >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 msgr2=0x7fe0f40d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0f9188640 1 -- 172.21.15.146:0/206514896 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507660345 0 0) 0x7fe0e4001110 con 0x7fe0f409bd80 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0fa18a640 1 --2- 172.21.15.146:0/206514896 >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 0x7fe0f40d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0fa18a640 1 -- 172.21.15.146:0/206514896 >> v1:172.21.15.177:6789/0 conn(0x7fe0f409bd80 legacy=0x7fe0f40ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0fa18a640 1 -- 172.21.15.146:0/206514896 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f40ceee0 con 0x7fe0f415e070 2024-04-25T15:21:49.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0fa98b640 1 --2- 172.21.15.146:0/206514896 >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 0x7fe0f40d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.614+0000 7fe0fa18a640 1 --2- 172.21.15.146:0/206514896 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cf9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009790 tx=0x7fe0f002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=844708a815f8cbca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0f9188640 1 -- 172.21.15.146:0/206514896 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0037070 con 0x7fe0f415e070 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0f9188640 1 -- 172.21.15.146:0/206514896 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f0032440 con 0x7fe0f415e070 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0f9188640 1 -- 172.21.15.146:0/206514896 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0032810 con 0x7fe0f415e070 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/206514896 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 msgr2=0x7fe0f40cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/206514896 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cf9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0009790 tx=0x7fe0f002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/206514896 shutdown_connections 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/206514896 >> v2:172.21.15.146:3300/0 conn(0x7fe0f40d2680 0x7fe0f40d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/206514896 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cf9c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/206514896 >> 172.21.15.146:0/206514896 conn(0x7fe0f40b7400 msgr2=0x7fe0f40b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/206514896 shutdown_connections 2024-04-25T15:21:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.615+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/206514896 wait complete. 2024-04-25T15:21:49.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.617+0000 7fe0fc3ee640 1 Processor -- start 2024-04-25T15:21:49.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.617+0000 7fe0fc3ee640 1 -- start start 2024-04-25T15:21:49.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.618+0000 7fe0fc3ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 0x7fe0f415d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.618+0000 7fe0fc3ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0f40d2680 0x7fe0f40ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.618+0000 7fe0fc3ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.618+0000 7fe0fc3ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f40cff00 con 0x7fe0f409bd80 2024-04-25T15:21:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.618+0000 7fe0fc3ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f40cf4a0 con 0x7fe0f40d2680 2024-04-25T15:21:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.618+0000 7fe0fc3ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe0f406cde0 con 0x7fe0f415e070 2024-04-25T15:21:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 0x7fe0f415d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 0x7fe0f415d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40438/0 (socket says 172.21.15.146:40438) 2024-04-25T15:21:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 -- 172.21.15.146:0/1615180807 learned_addr learned my addr 172.21.15.146:0/1615180807 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0f9989640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0f40d2680 0x7fe0f40ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa98b640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 -- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 msgr2=0x7fe0f40cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 -- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0f40d2680 msgr2=0x7fe0f40ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0f40d2680 0x7fe0f40ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 -- 172.21.15.146:0/1615180807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f40cb5f0 con 0x7fe0f409bd80 2024-04-25T15:21:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fa18a640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 0x7fe0f415d830 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0035040 tx=0x7fe0f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0042070 con 0x7fe0f409bd80 2024-04-25T15:21:49.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f0004220 con 0x7fe0f409bd80 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f40cb8d0 con 0x7fe0f409bd80 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0037040 con 0x7fe0f409bd80 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.619+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f40c9c50 con 0x7fe0f409bd80 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.620+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f40c9760 con 0x7fe0f409bd80 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.621+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fe0f0005990 con 0x7fe0f409bd80 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.621+0000 7fe0eaffd640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe0d8074570 0x7fe0d8076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.622+0000 7fe0f9989640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe0d8074570 0x7fe0d8076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.622+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe0f00b5f80 con 0x7fe0f409bd80 2024-04-25T15:21:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.622+0000 7fe0f9989640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe0d8074570 0x7fe0d8076a80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4000f60 tx=0x7fe0e4007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.626+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe0f007fe50 con 0x7fe0f409bd80 2024-04-25T15:21:49.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:49.840+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe0f406a0e0 con 0x7fe0f409bd80 2024-04-25T15:21:50.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.489+0000 7fe0eaffd640 1 -- 172.21.15.146:0/1615180807 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v19) ==== 162+0+0 (secure 0 0 0) 0x7fe0f0084150 con 0x7fe0f409bd80 2024-04-25T15:21:50.490 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-25T15:21:50.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe0d8074570 msgr2=0x7fe0d8076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe0d8074570 0x7fe0d8076a80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4000f60 tx=0x7fe0e4007e00 comp rx=0 tx=0).stop 2024-04-25T15:21:50.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 msgr2=0x7fe0f415d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 0x7fe0f415d830 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0035040 tx=0x7fe0f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 shutdown_connections 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe0d8074570 0x7fe0d8076a80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0f415e070 0x7fe0f40cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0f40d2680 0x7fe0f40ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 --2- 172.21.15.146:0/1615180807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0f409bd80 0x7fe0f415d830 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 >> 172.21.15.146:0/1615180807 conn(0x7fe0f40b7400 msgr2=0x7fe0f406e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 shutdown_connections 2024-04-25T15:21:50.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.491+0000 7fe0fc3ee640 1 -- 172.21.15.146:0/1615180807 wait complete. 2024-04-25T15:21:50.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:21:50.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.614+0000 7f052f9e3640 1 Processor -- start 2024-04-25T15:21:50.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.614+0000 7f052f9e3640 1 -- start start 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052f9e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806da30 0x7f052815e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052f9e3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f052806cf60 0x7f052806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052f9e3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05280b0f60 con 0x7f05280b0b50 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052f9e3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f05280b1790 con 0x7f052806cf60 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052f9e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f05280b2bb0 con 0x7f052806da30 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f052806cf60 0x7f052806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f052806cf60 0x7f052806d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50602/0 (socket says 172.21.15.146:50602) 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052cf7e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f05280b0b50 0x7f05280cc060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45116/0 (socket says 172.21.15.146:45116) 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 -- 172.21.15.146:0/467616839 learned_addr learned my addr 172.21.15.146:0/467616839 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052d77f640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806da30 0x7f052815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 -- 172.21.15.146:0/467616839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806da30 msgr2=0x7f052815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:50.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806da30 0x7f052815e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 -- 172.21.15.146:0/467616839 >> v1:172.21.15.146:6789/0 conn(0x7f05280b0b50 legacy=0x7f05280cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f051ffff640 1 -- 172.21.15.146:0/467616839 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242448368 0 0) 0x7f05280b0f60 con 0x7f05280b0b50 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052df80640 1 -- 172.21.15.146:0/467616839 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052806def0 con 0x7f052806cf60 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.616+0000 7f052d77f640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806da30 0x7f052815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.617+0000 7f052df80640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f052806cf60 0x7f052806d370 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0524009e50 tx=0x7f052402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1201501fe061dd5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.617+0000 7f051ffff640 1 -- 172.21.15.146:0/467616839 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524036070 con 0x7f052806cf60 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.617+0000 7f051ffff640 1 -- 172.21.15.146:0/467616839 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0524031600 con 0x7f052806cf60 2024-04-25T15:21:50.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.617+0000 7f051ffff640 1 -- 172.21.15.146:0/467616839 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05240319d0 con 0x7f052806cf60 2024-04-25T15:21:50.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 -- 172.21.15.146:0/467616839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f052806cf60 msgr2=0x7f052806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f052806cf60 0x7f052806d370 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f0524009e50 tx=0x7f052402a730 comp rx=0 tx=0).stop 2024-04-25T15:21:50.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 -- 172.21.15.146:0/467616839 shutdown_connections 2024-04-25T15:21:50.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f052806cf60 0x7f052806d370 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 --2- 172.21.15.146:0/467616839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806da30 0x7f052815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 -- 172.21.15.146:0/467616839 >> 172.21.15.146:0/467616839 conn(0x7f052809b2b0 msgr2=0x7f05280b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 -- 172.21.15.146:0/467616839 shutdown_connections 2024-04-25T15:21:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.618+0000 7f052f9e3640 1 -- 172.21.15.146:0/467616839 wait complete. 2024-04-25T15:21:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.620+0000 7f052f9e3640 1 Processor -- start 2024-04-25T15:21:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.620+0000 7f052f9e3640 1 -- start start 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052f9e3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806cf60 0x7f05281494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052f9e3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 0x7f0528149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052f9e3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0528155d50 0x7f05280afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052f9e3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f05280b2e50 con 0x7f05280b0b50 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052f9e3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05280b14b0 con 0x7f0528155d50 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052f9e3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f05280b3370 con 0x7f052806cf60 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 0x7f0528149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 0x7f0528149cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50616/0 (socket says 172.21.15.146:50616) 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 -- 172.21.15.146:0/2162904136 learned_addr learned my addr 172.21.15.146:0/2162904136 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052d77f640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806cf60 0x7f05281494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:50.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052df80640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0528155d50 0x7f05280afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 -- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806cf60 msgr2=0x7f05281494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806cf60 0x7f05281494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 -- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0528155d50 msgr2=0x7f05280afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0528155d50 0x7f05280afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052cf7e640 1 -- 172.21.15.146:0/2162904136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05281499e0 con 0x7f05280b0b50 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.621+0000 7f052d77f640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806cf60 0x7f05281494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.622+0000 7f052cf7e640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 0x7f0528149cc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f051800c3c0 tx=0x7f051800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.622+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05180074e0 con 0x7f05280b0b50 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.622+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0518007680 con 0x7f05280b0b50 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.622+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05280af1f0 con 0x7f05280b0b50 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.622+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518004a00 con 0x7f05280b0b50 2024-04-25T15:21:50.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.622+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0528156c10 con 0x7f05280b0b50 2024-04-25T15:21:50.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.623+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0528156680 con 0x7f05280b0b50 2024-04-25T15:21:50.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.624+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f0518011070 con 0x7f05280b0b50 2024-04-25T15:21:50.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.624+0000 7f051dffb640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f050c074540 0x7f050c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.624+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0518094f20 con 0x7f05280b0b50 2024-04-25T15:21:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.624+0000 7f052d77f640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f050c074540 0x7f050c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.625+0000 7f052d77f640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f050c074540 0x7f050c076a50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0520006fd0 tx=0x7f0520008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:50.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.629+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05281499e0 con 0x7f05280b0b50 2024-04-25T15:21:50.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.839+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f052806a0e0 con 0x7f05280b0b50 2024-04-25T15:21:50.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.840+0000 7f051dffb640 1 -- 172.21.15.146:0/2162904136 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5184 (secure 0 0 0) 0x7f051805f5e0 con 0x7f05280b0b50 2024-04-25T15:21:50.841 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:50.841 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:50.492308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T15:21:50.841 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 20 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f050c074540 msgr2=0x7f050c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f050c074540 0x7f050c076a50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0520006fd0 tx=0x7f0520008040 comp rx=0 tx=0).stop 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 msgr2=0x7f0528149cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 0x7f0528149cc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f051800c3c0 tx=0x7f051800c880 comp rx=0 tx=0).stop 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 shutdown_connections 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f050c074540 0x7f050c076a50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0528155d50 0x7f05280afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05280b0b50 0x7f0528149cc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 --2- 172.21.15.146:0/2162904136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f052806cf60 0x7f05281494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:50.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 >> 172.21.15.146:0/2162904136 conn(0x7f052809b2b0 msgr2=0x7f05281575c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:50.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 shutdown_connections 2024-04-25T15:21:50.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:50.842+0000 7f052f9e3640 1 -- 172.21.15.146:0/2162904136 wait complete. 2024-04-25T15:21:51.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:21:07.618988+0000', 'modified': '2024-04-25T15:21:50.492308+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5553': {'gid': 5553, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.177:6834/377672985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 377672985}, {'type': 'v1', 'addr': '172.21.15.177:6834', 'nonce': 377672985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T15:21:51.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5553:up:replay 2024-04-25T15:21:52.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:21:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.385+0000 7fd61a696640 1 Processor -- start 2024-04-25T15:21:52.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.385+0000 7fd61a696640 1 -- start start 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd61a696640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 0x7fd614071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd61a696640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 0x7fd614161ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd61a696640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6140cfb60 con 0x7fd6140cf040 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd61a696640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd6140d2a20 con 0x7fd614161ab0 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd61a696640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd6140d2460 con 0x7fd6140b8dc0 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd613fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 0x7fd614071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd618c33640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 0x7fd614161ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:52.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd613fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 0x7fd614071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:40212/0 (socket says 172.21.15.146:40212) 2024-04-25T15:21:52.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd618c33640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 0x7fd614161ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:41760/0 (socket says 172.21.15.146:41760) 2024-04-25T15:21:52.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd613fff640 1 -- 172.21.15.146:0/2419248898 learned_addr learned my addr 172.21.15.146:0/2419248898 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:52.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.386+0000 7fd6137fe640 1 --1- 172.21.15.146:0/2419248898 >> v1:172.21.15.177:6789/0 conn(0x7fd6140cf040 0x7fd6140cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:39776/0 (socket says 172.21.15.146:39776) 2024-04-25T15:21:52.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.387+0000 7fd613fff640 1 -- 172.21.15.146:0/2419248898 >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 msgr2=0x7fd614161ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:52.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.387+0000 7fd613fff640 1 --2- 172.21.15.146:0/2419248898 >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 0x7fd614161ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.387+0000 7fd613fff640 1 -- 172.21.15.146:0/2419248898 >> v1:172.21.15.177:6789/0 conn(0x7fd6140cf040 legacy=0x7fd6140cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:52.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.387+0000 7fd613fff640 1 -- 172.21.15.146:0/2419248898 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6140d2740 con 0x7fd6140b8dc0 2024-04-25T15:21:52.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.387+0000 7fd612ffd640 1 -- 172.21.15.146:0/2419248898 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160669183 0 0) 0x7fd6140cfb60 con 0x7fd6140cf040 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.387+0000 7fd618c33640 1 --2- 172.21.15.146:0/2419248898 >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 0x7fd614161ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.388+0000 7fd613fff640 1 --2- 172.21.15.146:0/2419248898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 0x7fd614071620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd604009880 tx=0x7fd60402a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d4a89797be74618 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.388+0000 7fd612ffd640 1 -- 172.21.15.146:0/2419248898 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604004030 con 0x7fd6140b8dc0 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.388+0000 7fd612ffd640 1 -- 172.21.15.146:0/2419248898 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6040041d0 con 0x7fd6140b8dc0 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.388+0000 7fd612ffd640 1 -- 172.21.15.146:0/2419248898 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604003610 con 0x7fd6140b8dc0 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.388+0000 7fd61a696640 1 -- 172.21.15.146:0/2419248898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 msgr2=0x7fd614071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.388+0000 7fd61a696640 1 --2- 172.21.15.146:0/2419248898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 0x7fd614071620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd604009880 tx=0x7fd60402a200 comp rx=0 tx=0).stop 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.389+0000 7fd61a696640 1 -- 172.21.15.146:0/2419248898 shutdown_connections 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.389+0000 7fd61a696640 1 --2- 172.21.15.146:0/2419248898 >> v2:172.21.15.177:3300/0 conn(0x7fd614161ab0 0x7fd614161ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.389+0000 7fd61a696640 1 --2- 172.21.15.146:0/2419248898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140b8dc0 0x7fd614071620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.389+0000 7fd61a696640 1 -- 172.21.15.146:0/2419248898 >> 172.21.15.146:0/2419248898 conn(0x7fd6140b5360 msgr2=0x7fd6140b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.389+0000 7fd61a696640 1 -- 172.21.15.146:0/2419248898 shutdown_connections 2024-04-25T15:21:52.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.389+0000 7fd61a696640 1 -- 172.21.15.146:0/2419248898 wait complete. 2024-04-25T15:21:52.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.390+0000 7fd61a696640 1 Processor -- start 2024-04-25T15:21:52.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.391+0000 7fd61a696640 1 -- start start 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd61a696640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6140b8dc0 0x7fd614151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd61a696640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140cf040 0x7fd6140ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd61a696640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 0x7fd6140cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd61a696640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd614071b60 con 0x7fd614161ab0 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd61a696640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd6140ce4f0 con 0x7fd6140b8dc0 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd61a696640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd6140d20f0 con 0x7fd6140cf040 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 0x7fd6140cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 0x7fd6140cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50632/0 (socket says 172.21.15.146:50632) 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 -- 172.21.15.146:0/3145622385 learned_addr learned my addr 172.21.15.146:0/3145622385 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd6137fe640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140cf040 0x7fd6140ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:52.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd613fff640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6140b8dc0 0x7fd614151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 -- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140cf040 msgr2=0x7fd6140ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140cf040 0x7fd6140ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 -- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6140b8dc0 msgr2=0x7fd614151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6140b8dc0 0x7fd614151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.392+0000 7fd618c33640 1 -- 172.21.15.146:0/3145622385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6140b6aa0 con 0x7fd614161ab0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.393+0000 7fd618c33640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 0x7fd6140cd1e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009e00 tx=0x7fd60c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.393+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c018020 con 0x7fd614161ab0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.393+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60c014480 con 0x7fd614161ab0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.393+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6141520f0 con 0x7fd614161ab0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.393+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c014870 con 0x7fd614161ab0 2024-04-25T15:21:52.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.393+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6140b7060 con 0x7fd614161ab0 2024-04-25T15:21:52.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.394+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61415aea0 con 0x7fd614161ab0 2024-04-25T15:21:52.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.395+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fd60c005d10 con 0x7fd614161ab0 2024-04-25T15:21:52.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.395+0000 7fd610ff9640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd600074570 0x7fd600076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:52.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.395+0000 7fd613fff640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd600074570 0x7fd600076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:52.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.395+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd60c096010 con 0x7fd614161ab0 2024-04-25T15:21:52.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.396+0000 7fd613fff640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd600074570 0x7fd600076a80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd604033000 tx=0x7fd60402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:52.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.400+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd60c022030 con 0x7fd614161ab0 2024-04-25T15:21:52.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.611+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6140b72a0 con 0x7fd614161ab0 2024-04-25T15:21:52.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.612+0000 7fd610ff9640 1 -- 172.21.15.146:0/3145622385 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5185 (secure 0 0 0) 0x7fd60c010040 con 0x7fd614161ab0 2024-04-25T15:21:52.613 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:51.502468+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T15:21:52.613 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 22 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd600074570 msgr2=0x7fd600076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd600074570 0x7fd600076a80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd604033000 tx=0x7fd60402e000 comp rx=0 tx=0).stop 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 msgr2=0x7fd6140cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 0x7fd6140cd1e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009e00 tx=0x7fd60c00c490 comp rx=0 tx=0).stop 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 shutdown_connections 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd600074570 0x7fd600076a80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd614161ab0 0x7fd6140cd1e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6140cf040 0x7fd6140ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 --2- 172.21.15.146:0/3145622385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6140b8dc0 0x7fd614151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:52.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 >> 172.21.15.146:0/3145622385 conn(0x7fd6140b5360 msgr2=0x7fd6140cf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:52.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 shutdown_connections 2024-04-25T15:21:52.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:52.614+0000 7fd61a696640 1 -- 172.21.15.146:0/3145622385 wait complete. 2024-04-25T15:21:53.028 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:21:07.618988+0000', 'modified': '2024-04-25T15:21:51.502468+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5553': {'gid': 5553, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.177:6834/377672985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 377672985}, {'type': 'v1', 'addr': '172.21.15.177:6834', 'nonce': 377672985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T15:21:53.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5553:up:rejoin 2024-04-25T15:21:54.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:21:54.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.139+0000 7f1b09f7a640 1 Processor -- start 2024-04-25T15:21:54.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.139+0000 7f1b09f7a640 1 -- start start 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b09f7a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1b040cbc90 0x7f1b04153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b09f7a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b040d0ac0 0x7f1b040d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b09f7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b040d1410 con 0x7f1b040d01a0 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b09f7a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f1b041541e0 con 0x7f1b040cbc90 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b09f7a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1b040cfd10 con 0x7f1b040d0ac0 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b02ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1b040d01a0 0x7f1b0415dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45124/0 (socket says 172.21.15.146:45124) 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b02ffd640 1 -- 172.21.15.146:0/2209843313 learned_addr learned my addr 172.21.15.146:0/2209843313 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b03fff640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b040d0ac0 0x7f1b040d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b037fe640 1 --2- 172.21.15.146:0/2209843313 >> v2:172.21.15.177:3300/0 conn(0x7f1b040cbc90 0x7f1b04153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3284208125 0 0) 0x7f1af8001110 con 0x7f1b040d01a0 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1aec003520 con 0x7f1b040d01a0 2024-04-25T15:21:54.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.141+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 845103664 0 0) 0x7f1aec003520 con 0x7f1b040d01a0 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b040d0ac0 msgr2=0x7f1b040d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b040d0ac0 0x7f1b040d0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 >> v2:172.21.15.177:3300/0 conn(0x7f1b040cbc90 msgr2=0x7f1b04153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 --2- 172.21.15.146:0/2209843313 >> v2:172.21.15.177:3300/0 conn(0x7f1b040cbc90 0x7f1b04153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b040cf420 con 0x7f1b040d01a0 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f1af8001840 con 0x7f1b040d01a0 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b037fe640 1 --2- 172.21.15.146:0/2209843313 >> v2:172.21.15.177:3300/0 conn(0x7f1b040cbc90 0x7f1b04153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b040d0ac0 0x7f1b040d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 >> v1:172.21.15.146:6789/0 conn(0x7f1b040d01a0 legacy=0x7f1b0415dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aec0039b0 0x7f1aec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec004800 0x7f1aec004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec005150 0x7f1aec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1b040cfd10 con 0x7f1aec005150 2024-04-25T15:21:54.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec00e110 con 0x7f1aec0039b0 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec00e310 con 0x7f1aec004800 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec005150 0x7f1aec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b02ffd640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec004800 0x7f1aec004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b037fe640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aec0039b0 0x7f1aec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 -- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec004800 msgr2=0x7f1aec004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec004800 0x7f1aec004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 -- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aec0039b0 msgr2=0x7f1aec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aec0039b0 0x7f1aec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.142+0000 7f1b03fff640 1 -- 172.21.15.146:0/2209843313 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec003520 con 0x7f1aec005150 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.143+0000 7f1b03fff640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec005150 0x7f1aec00b9c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1b040b0b20 tx=0x7f1aec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bce3822f08a7f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.143+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc036070 con 0x7f1aec005150 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.143+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1afc003b90 con 0x7f1aec005150 2024-04-25T15:21:54.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.143+0000 7f1b027fc640 1 -- 172.21.15.146:0/2209843313 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc009d60 con 0x7f1aec005150 2024-04-25T15:21:54.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b09f7a640 1 -- 172.21.15.146:0/2209843313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec005150 msgr2=0x7f1aec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/2209843313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec005150 0x7f1aec00b9c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1b040b0b20 tx=0x7f1aec005830 comp rx=0 tx=0).stop 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b03fff640 1 -- 172.21.15.146:0/2209843313 reap_dead start 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b09f7a640 1 -- 172.21.15.146:0/2209843313 shutdown_connections 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b09f7a640 1 -- 172.21.15.146:0/2209843313 >> 172.21.15.146:0/2209843313 conn(0x7f1b040ce0c0 msgr2=0x7f1b040ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b09f7a640 1 -- 172.21.15.146:0/2209843313 shutdown_connections 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.144+0000 7f1b09f7a640 1 -- 172.21.15.146:0/2209843313 wait complete. 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.145+0000 7f1b09f7a640 1 Processor -- start 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.146+0000 7f1b09f7a640 1 -- start start 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b09f7a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec005150 0x7f1b040b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b09f7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 0x7f1b040b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b09f7a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04145df0 0x7f1b041f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b09f7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1b040cf9a0 con 0x7f1b041459e0 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b09f7a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b040cf6c0 con 0x7f1b04145df0 2024-04-25T15:21:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b09f7a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1b040d05b0 con 0x7f1aec005150 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 0x7f1b040b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 0x7f1b040b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50640/0 (socket says 172.21.15.146:50640) 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 -- 172.21.15.146:0/539866482 learned_addr learned my addr 172.21.15.146:0/539866482 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b037fe640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec005150 0x7f1b040b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b03fff640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04145df0 0x7f1b041f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 -- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec005150 msgr2=0x7f1b040b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec005150 0x7f1b040b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 -- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04145df0 msgr2=0x7f1b041f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04145df0 0x7f1b041f2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b02ffd640 1 -- 172.21.15.146:0/539866482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b040b5ac0 con 0x7f1b041459e0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.147+0000 7f1b037fe640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec005150 0x7f1b040b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.148+0000 7f1b02ffd640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 0x7f1b040b4eb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1af8007310 tx=0x7f1af80077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.148+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af800f9c0 con 0x7f1b041459e0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.148+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af8018070 con 0x7f1b041459e0 2024-04-25T15:21:54.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.148+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b040b5da0 con 0x7f1b041459e0 2024-04-25T15:21:54.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.148+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af800fdc0 con 0x7f1b041459e0 2024-04-25T15:21:54.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.148+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b040b6360 con 0x7f1b041459e0 2024-04-25T15:21:54.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.149+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b040b5830 con 0x7f1b041459e0 2024-04-25T15:21:54.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.149+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f1af801d460 con 0x7f1b041459e0 2024-04-25T15:21:54.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.150+0000 7f1aebfff640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1af0074540 0x7f1af0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.150+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f1af8095560 con 0x7f1b041459e0 2024-04-25T15:21:54.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.150+0000 7f1b037fe640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1af0074540 0x7f1af0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.150+0000 7f1b037fe640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1af0074540 0x7f1af0076a50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b0406c600 tx=0x7f1af4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.154+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b040d1410 con 0x7f1b041459e0 2024-04-25T15:21:54.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.364+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b0406a0e0 con 0x7f1b041459e0 2024-04-25T15:21:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.365+0000 7f1aebfff640 1 -- 172.21.15.146:0/539866482 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5192 (secure 0 0 0) 0x7f1af805fb90 con 0x7f1b041459e0 2024-04-25T15:21:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:54.366 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:53.503394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5553,"qdb_cluster":[5553]},"id":2}]} 2024-04-25T15:21:54.366 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1af0074540 msgr2=0x7f1af0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1af0074540 0x7f1af0076a50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b0406c600 tx=0x7f1af4008040 comp rx=0 tx=0).stop 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 msgr2=0x7f1b040b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 0x7f1b040b4eb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1af8007310 tx=0x7f1af80077d0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 shutdown_connections 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1af0074540 0x7f1af0076a50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b04145df0 0x7f1b041f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b041459e0 0x7f1b040b4eb0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 --2- 172.21.15.146:0/539866482 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1aec005150 0x7f1b040b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 >> 172.21.15.146:0/539866482 conn(0x7f1b040ce0c0 msgr2=0x7f1b0406f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 shutdown_connections 2024-04-25T15:21:54.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.367+0000 7f1b09f7a640 1 -- 172.21.15.146:0/539866482 wait complete. 2024-04-25T15:21:54.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:21:07.618988+0000', 'modified': '2024-04-25T15:21:53.503394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5553': {'gid': 5553, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.177:6834/377672985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 377672985}, {'type': 'v1', 'addr': '172.21.15.177:6834', 'nonce': 377672985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5553, 'qdb_cluster': [5553]} 2024-04-25T15:21:54.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:21:54.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T15:21:54.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.891+0000 7f3b7cdce640 1 Processor -- start 2024-04-25T15:21:54.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.891+0000 7f3b7cdce640 1 -- start start 2024-04-25T15:21:54.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7cdce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3b78153ca0 0x7f3b7815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7cdce640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b780cf420 con 0x7f3b780cbc90 2024-04-25T15:21:54.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7cdce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b780cc270 con 0x7f3b780d0ac0 2024-04-25T15:21:54.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7cdce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3b780cfd10 con 0x7f3b78153ca0 2024-04-25T15:21:54.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b75d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3b78153ca0 0x7f3b7815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3b780cbc90 0x7f3b780d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45136/0 (socket says 172.21.15.146:45136) 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b75d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3b78153ca0 0x7f3b7815dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50656/0 (socket says 172.21.15.146:50656) 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7659c640 1 -- 172.21.15.146:0/3342718299 learned_addr learned my addr 172.21.15.146:0/3342718299 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7559a640 1 -- 172.21.15.146:0/3342718299 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190332889 0 0) 0x7f3b780cf420 con 0x7f3b780cbc90 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b7559a640 1 -- 172.21.15.146:0/3342718299 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b60003520 con 0x7f3b780cbc90 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b75d9b640 1 -- 172.21.15.146:0/3342718299 >> v1:172.21.15.177:6789/0 conn(0x7f3b780d0ac0 legacy=0x7f3b780d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b75d9b640 1 -- 172.21.15.146:0/3342718299 >> v1:172.21.15.146:6789/0 conn(0x7f3b780cbc90 legacy=0x7f3b780d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.893+0000 7f3b75d9b640 1 -- 172.21.15.146:0/3342718299 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b780cf6c0 con 0x7f3b78153ca0 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.894+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/3342718299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b7815dc60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0097a0 tx=0x7f3b6c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49590e52c613ad2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.894+0000 7f3b7559a640 1 -- 172.21.15.146:0/3342718299 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c037070 con 0x7f3b78153ca0 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.894+0000 7f3b7559a640 1 -- 172.21.15.146:0/3342718299 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6c032440 con 0x7f3b78153ca0 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.894+0000 7f3b7559a640 1 -- 172.21.15.146:0/3342718299 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c032810 con 0x7f3b78153ca0 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 -- 172.21.15.146:0/3342718299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 msgr2=0x7f3b7815dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/3342718299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b7815dc60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0097a0 tx=0x7f3b6c02a230 comp rx=0 tx=0).stop 2024-04-25T15:21:54.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 -- 172.21.15.146:0/3342718299 shutdown_connections 2024-04-25T15:21:54.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/3342718299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b7815dc60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 -- 172.21.15.146:0/3342718299 >> 172.21.15.146:0/3342718299 conn(0x7f3b780ce0c0 msgr2=0x7f3b780ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 -- 172.21.15.146:0/3342718299 shutdown_connections 2024-04-25T15:21:54.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.895+0000 7f3b7cdce640 1 -- 172.21.15.146:0/3342718299 wait complete. 2024-04-25T15:21:54.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.897+0000 7f3b7cdce640 1 Processor -- start 2024-04-25T15:21:54.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.897+0000 7f3b7cdce640 1 -- start start 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7cdce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b780cbc90 0x7f3b78150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7cdce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b780d0ac0 0x7f3b781f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7cdce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b780b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7cdce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3b781540b0 con 0x7f3b78153ca0 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7cdce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7815e1a0 con 0x7f3b780cbc90 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7cdce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3b78070390 con 0x7f3b780d0ac0 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b76d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b780b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b76d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b780b6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50666/0 (socket says 172.21.15.146:50666) 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b76d9d640 1 -- 172.21.15.146:0/501315586 learned_addr learned my addr 172.21.15.146:0/501315586 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b7659c640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b780cbc90 0x7f3b78150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.898+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b780d0ac0 0x7f3b781f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b76d9d640 1 -- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b780d0ac0 msgr2=0x7f3b781f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b780d0ac0 0x7f3b781f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b76d9d640 1 -- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b780cbc90 msgr2=0x7f3b78150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b780cbc90 0x7f3b78150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b76d9d640 1 -- 172.21.15.146:0/501315586 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b781575d0 con 0x7f3b78153ca0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b780b6610 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000c760 tx=0x7f3b7000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b66ffd640 1 -- 172.21.15.146:0/501315586 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70016020 con 0x7f3b78153ca0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b66ffd640 1 -- 172.21.15.146:0/501315586 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b70006d50 con 0x7f3b78153ca0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b781f5330 con 0x7f3b78153ca0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b66ffd640 1 -- 172.21.15.146:0/501315586 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70003bb0 con 0x7f3b78153ca0 2024-04-25T15:21:54.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.899+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b780afae0 con 0x7f3b78153ca0 2024-04-25T15:21:54.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.901+0000 7f3b66ffd640 1 -- 172.21.15.146:0/501315586 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f3b7000a850 con 0x7f3b78153ca0 2024-04-25T15:21:54.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.901+0000 7f3b66ffd640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3b58074540 0x7f3b58076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.901+0000 7f3b7659c640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3b58074540 0x7f3b58076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.901+0000 7f3b66ffd640 1 -- 172.21.15.146:0/501315586 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f3b70094e80 con 0x7f3b78153ca0 2024-04-25T15:21:54.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.902+0000 7f3b7659c640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3b58074540 0x7f3b58076a50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3b680011e0 tx=0x7f3b68007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.902+0000 7f3b83c81740 1 Processor -- start 2024-04-25T15:21:54.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.902+0000 7f3b83c81740 1 -- start start 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.903+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1ff270 0x55d66c5cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.903+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 0x55d66c5c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.903+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5a1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.903+0000 7f3b83c81740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d66c505740 con 0x55d66c1d63b0 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.903+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d66c4a13a0 con 0x55d66c1ff270 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.903+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55d66c501700 con 0x55d66c1de590 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 0x55d66c5c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 0x55d66c5c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50670/0 (socket says 172.21.15.146:50670) 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 -- 172.21.15.146:0/1346931824 learned_addr learned my addr 172.21.15.146:0/1346931824 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b7659c640 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1ff270 0x55d66c5cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 -- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 msgr2=0x55d66c5a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 -- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1ff270 msgr2=0x55d66c5cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1ff270 0x55d66c5cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 -- 172.21.15.146:0/1346931824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d66c4bd750 con 0x55d66c1d63b0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 0x55d66c5c7fa0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c000c00 tx=0x7f3b6c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f146b8606ece7705 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b57fff640 1 -- 172.21.15.146:0/1346931824 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c037040 con 0x55d66c1d63b0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b57fff640 1 -- 172.21.15.146:0/1346931824 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6c032800 con 0x55d66c1d63b0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.904+0000 7f3b57fff640 1 -- 172.21.15.146:0/1346931824 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c032bf0 con 0x55d66c1d63b0 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 -- 172.21.15.146:0/1346931824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 msgr2=0x55d66c5c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 0x55d66c5c7fa0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c000c00 tx=0x7f3b6c02c000 comp rx=0 tx=0).stop 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 -- 172.21.15.146:0/1346931824 shutdown_connections 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1d63b0 0x55d66c5c7fa0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1346931824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1ff270 0x55d66c5cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 -- 172.21.15.146:0/1346931824 >> 172.21.15.146:0/1346931824 conn(0x55d66c1f33c0 msgr2=0x55d66c5a09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 -- 172.21.15.146:0/1346931824 shutdown_connections 2024-04-25T15:21:54.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.905+0000 7f3b83c81740 1 -- 172.21.15.146:0/1346931824 wait complete. 2024-04-25T15:21:54.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.906+0000 7f3b83c81740 1 -- start start 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.906+0000 7f3b83c81740 1 Processor -- start 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d66c48cfe0 con 0x55d66c1ff270 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d66c5c9200 con 0x55d66c1d63b0 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55d66c5c5dd0 con 0x55d66c1de590 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 10 client.0 _ll_register_callbacks cb 0x55d66c4a3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50680/0 (socket says 172.21.15.146:50680) 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 -- 172.21.15.146:0/551283999 learned_addr learned my addr 172.21.15.146:0/551283999 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b7659c640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 -- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 msgr2=0x55d66c5cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 -- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 msgr2=0x55d66c5b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 -- 172.21.15.146:0/551283999 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d66c2172d0 con 0x55d66c1ff270 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b7659c640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c8370 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3b7005b460 tx=0x7f3b7005ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2faed71840c6bad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b557fa640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b557fa640 1 -- 172.21.15.146:0/551283999 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7007d060 con 0x55d66c1ff270 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b557fa640 1 -- 172.21.15.146:0/551283999 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f3b7000ad40 con 0x55d66c1ff270 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.908+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b6c032800 con 0x55d66c1ff270 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b557fa640 1 -- 172.21.15.146:0/551283999 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3b70091050 con 0x55d66c1ff270 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 10 client.6096 fetch_fsmap learned FSMap version 23 2024-04-25T15:21:54.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55d66c2e16b0 con 0x55d66c1ff270 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b557fa640 1 -- 172.21.15.146:0/551283999 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6c032800 con 0x55d66c1ff270 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b557fa640 1 -- 172.21.15.146:0/551283999 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f3b7005e0d0 con 0x55d66c1ff270 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 10 client.6096 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 10 client.6096 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 20 client.6096 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c1bde40 0x55d66c602ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 4 client.6096 mds_command: new command op to 5553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 --> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d66c513350 con 0x55d66c1bde40 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.909+0000 7f3b7659c640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c1bde40 0x55d66c602ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.910+0000 7f3b557fa640 10 client.6096 ms_handle_connect on v2:172.21.15.177:6832/377672985 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.910+0000 7f3b7659c640 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c1bde40 0x55d66c602ae0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8de25a1a578bf83e server_cookie=b5b0fc9f96716c2a in_seq=0 out_seq=0 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.910+0000 7f3b557fa640 1 -- 172.21.15.146:0/551283999 <== mds.0 v2:172.21.15.177:6832/377672985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d66c513350 con 0x55d66c1bde40 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b557fa640 10 client.6096 handle_command_reply: tid=0 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 client.6096 shutdown 2024-04-25T15:21:54.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 msgr2=0x55d66c5c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c8370 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3b7005b460 tx=0x7f3b7005ead0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 shutdown_connections 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c1bde40 0x55d66c602ae0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c8370 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c5cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 --2- 172.21.15.146:0/551283999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b557fa640 0 client.6096 ms_handle_reset on v2:172.21.15.177:6832/377672985 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 >> 172.21.15.146:0/551283999 conn(0x55d66c1f33c0 msgr2=0x55d66c5a09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 shutdown_connections 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 1 -- 172.21.15.146:0/551283999 wait complete. 2024-04-25T15:21:54.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.911+0000 7f3b83c81740 20 client.6096 trim_cache size 0 max 16384 2024-04-25T15:21:54.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.930+0000 7f3b83c81740 1 Processor -- start 2024-04-25T15:21:54.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.930+0000 7f3b83c81740 1 -- start start 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c59a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b83c81740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3b78200fb0 con 0x55d66c1ff270 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b78200040 con 0x55d66c1d63b0 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7806d6f0 con 0x55d66c1de590 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50688/0 (socket says 172.21.15.146:50688) 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 -- 172.21.15.146:0/1852187869 learned_addr learned my addr 172.21.15.146:0/1852187869 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b7659c640 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c59a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 -- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 msgr2=0x55d66c59a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c59a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 -- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 msgr2=0x55d66c5ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.931+0000 7f3b76d9d640 1 -- 172.21.15.146:0/1852187869 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d66c59c1b0 con 0x55d66c1ff270 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.932+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c75c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3b700951f0 tx=0x7f3b70080070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99107541aac513a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.932+0000 7f3b577fe640 1 -- 172.21.15.146:0/1852187869 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70084070 con 0x55d66c1ff270 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.932+0000 7f3b577fe640 1 -- 172.21.15.146:0/1852187869 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b70075070 con 0x55d66c1ff270 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.932+0000 7f3b577fe640 1 -- 172.21.15.146:0/1852187869 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7005b4c0 con 0x55d66c1ff270 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.932+0000 7f3b83c81740 1 -- 172.21.15.146:0/1852187869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 msgr2=0x55d66c5c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.932+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c75c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3b700951f0 tx=0x7f3b70080070 comp rx=0 tx=0).stop 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 -- 172.21.15.146:0/1852187869 shutdown_connections 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1ff270 0x55d66c5c75c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1de590 0x55d66c59a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 --2- 172.21.15.146:0/1852187869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 -- 172.21.15.146:0/1852187869 >> 172.21.15.146:0/1852187869 conn(0x55d66c1f33c0 msgr2=0x55d66c5e51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 -- 172.21.15.146:0/1852187869 shutdown_connections 2024-04-25T15:21:54.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 -- 172.21.15.146:0/1852187869 wait complete. 2024-04-25T15:21:54.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 -- start start 2024-04-25T15:21:54.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.933+0000 7f3b83c81740 1 Processor -- start 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 0x55d66c5e8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1ff270 0x55d66c5e8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d66c5c5c30 con 0x55d66c1de590 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b781ffbb0 con 0x55d66c1d63b0 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55d66c5ce9e0 con 0x55d66c1ff270 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b83c81740 10 client.0 _ll_register_callbacks cb 0x7f3b70001710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 0x55d66c5e8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 0x55d66c5e8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50702/0 (socket says 172.21.15.146:50702) 2024-04-25T15:21:54.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 -- 172.21.15.146:0/22046827 learned_addr learned my addr 172.21.15.146:0/22046827 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b7659c640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b76d9d640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1ff270 0x55d66c5e8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 -- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1ff270 msgr2=0x55d66c5e8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1ff270 0x55d66c5e8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 -- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 msgr2=0x55d66c5a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.935+0000 7f3b75d9b640 1 -- 172.21.15.146:0/22046827 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d66c5ea2b0 con 0x55d66c1de590 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b75d9b640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 0x55d66c5e8540 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0098c0 tx=0x7f3b6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac5b6ec29c4b2a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b557fa640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b557fa640 1 -- 172.21.15.146:0/22046827 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c04a070 con 0x55d66c1de590 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b557fa640 1 -- 172.21.15.146:0/22046827 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f3b6c003a20 con 0x55d66c1de590 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b70075070 con 0x55d66c1de590 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b557fa640 1 -- 172.21.15.146:0/22046827 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3b6c055050 con 0x55d66c1de590 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 10 client.6102 fetch_fsmap learned FSMap version 23 2024-04-25T15:21:54.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55d66c668570 con 0x55d66c1de590 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b557fa640 1 -- 172.21.15.146:0/22046827 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b70075070 con 0x55d66c1de590 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b557fa640 1 -- 172.21.15.146:0/22046827 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f3b6c058020 con 0x55d66c1de590 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 10 client.6102 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 10 client.6102 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 20 client.6102 populate_metadata read hostname 'smithi146' 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c5f6f00 0x55d66c5f7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 4 client.6102 mds_command: new command op to 5553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 --> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] -- command(tid 0: {"prefix": "status"}) -- 0x55d66c50d1c0 con 0x55d66c5f6f00 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.936+0000 7f3b7659c640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c5f6f00 0x55d66c5f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b557fa640 10 client.6102 ms_handle_connect on v2:172.21.15.177:6832/377672985 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b7659c640 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c5f6f00 0x55d66c5f7310 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d5910410f12a461 server_cookie=9eedf67135e7bfe0 in_seq=0 out_seq=0 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b557fa640 1 -- 172.21.15.146:0/22046827 <== mds.0 v2:172.21.15.177:6832/377672985 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d66c50d1c0 con 0x55d66c5f6f00 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b557fa640 10 client.6102 handle_command_reply: tid=0 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 client.6102 shutdown 2024-04-25T15:21:54.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 msgr2=0x55d66c5e8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 0x55d66c5e8540 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0098c0 tx=0x7f3b6c004730 comp rx=0 tx=0).stop 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 shutdown_connections 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:6832/377672985,v1:172.21.15.177:6834/377672985] conn(0x55d66c5f6f00 0x55d66c5f7310 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d66c1ff270 0x55d66c5e8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d66c1de590 0x55d66c5e8540 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 --2- 172.21.15.146:0/22046827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d66c1d63b0 0x55d66c5a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b557fa640 0 client.6102 ms_handle_reset on v2:172.21.15.177:6832/377672985 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 >> 172.21.15.146:0/22046827 conn(0x55d66c1f33c0 msgr2=0x55d66c5cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 shutdown_connections 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.937+0000 7f3b83c81740 1 -- 172.21.15.146:0/22046827 wait complete. 2024-04-25T15:21:54.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b83c81740 20 client.6102 trim_cache size 0 max 16384 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "c6e9a678-3722-48f4-a297-70f509c13e29", 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5553, 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-25T15:21:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 4.4412424809999997, 2024-04-25T15:21:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-25T15:21:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 42, 2024-04-25T15:21:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 42, 2024-04-25T15:21:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 53.538524674999998 2024-04-25T15:21:54.945 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3b58074540 msgr2=0x7f3b58076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3b58074540 0x7f3b58076a50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3b680011e0 tx=0x7f3b68007d30 comp rx=0 tx=0).stop 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 msgr2=0x7f3b780b6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b780b6610 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000c760 tx=0x7f3b7000cc20 comp rx=0 tx=0).stop 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 shutdown_connections 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3b58074540 0x7f3b58076a50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b78153ca0 0x7f3b780b6610 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b780d0ac0 0x7f3b781f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 --2- 172.21.15.146:0/501315586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b780cbc90 0x7f3b78150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:54.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 >> 172.21.15.146:0/501315586 conn(0x7f3b780ce0c0 msgr2=0x7f3b780b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:54.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 shutdown_connections 2024-04-25T15:21:54.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:54.938+0000 7f3b7cdce640 1 -- 172.21.15.146:0/501315586 wait complete. 2024-04-25T15:21:54.950 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:21:54.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:21:54.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:21:55.036 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:21:55.036 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:21:55.036 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:55.036 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:55.051 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:55.052 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:55.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.253+0000 7f68c42aa640 1 Processor -- start 2024-04-25T15:21:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.253+0000 7f68c42aa640 1 -- start start 2024-04-25T15:21:55.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c42aa640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68bc0d0730 0x7f68bc0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c42aa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68bc0b9200 con 0x7f68bc06ccc0 2024-04-25T15:21:55.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c42aa640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68bc0d1070 con 0x7f68bc0d4f60 2024-04-25T15:21:55.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c42aa640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc06c550 con 0x7f68bc0d0730 2024-04-25T15:21:55.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c2847640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f68bc06ccc0 0x7f68bc06d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45142/0 (socket says 172.21.15.146:45142) 2024-04-25T15:21:55.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c2847640 1 -- 172.21.15.146:0/957455845 learned_addr learned my addr 172.21.15.146:0/957455845 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419575702 0 0) 0x7f68b4001110 con 0x7f68bc06ccc0 2024-04-25T15:21:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1845640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68bc0d0730 0x7f68bc0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a8003520 con 0x7f68bc06ccc0 2024-04-25T15:21:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2454007753 0 0) 0x7f68a8003520 con 0x7f68bc06ccc0 2024-04-25T15:21:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68bc0d0730 msgr2=0x7f68bc0d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68bc0d0730 0x7f68bc0d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 >> v1:172.21.15.177:6789/0 conn(0x7f68bc0d4f60 legacy=0x7f68bc0d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc0d1310 con 0x7f68bc06ccc0 2024-04-25T15:21:55.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.255+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f68b4001840 con 0x7f68bc06ccc0 2024-04-25T15:21:55.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 >> v1:172.21.15.146:6789/0 conn(0x7f68bc06ccc0 legacy=0x7f68bc06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a80039e0 0x7f68a8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8004370 0x7f68a80048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a8005160 0x7f68a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc06c550 con 0x7f68a8004370 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f68a800c090 con 0x7f68a8005160 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f68a800c290 con 0x7f68a80039e0 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1845640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8004370 0x7f68a80048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c2046640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a80039e0 0x7f68a8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c2847640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a8005160 0x7f68a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1845640 1 -- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a80039e0 msgr2=0x7f68a8003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1845640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a80039e0 0x7f68a8003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1845640 1 -- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a8005160 msgr2=0x7f68a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1845640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a8005160 0x7f68a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c1845640 1 -- 172.21.15.146:0/957455845 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a8003520 con 0x7f68a8004370 2024-04-25T15:21:55.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.256+0000 7f68c2847640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68a8005160 0x7f68a800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c1845640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8004370 0x7f68a80048c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009ac0 tx=0x7f68a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436684f871aea36b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac036070 con 0x7f68a8004370 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ac031400 con 0x7f68a8004370 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c1044640 1 -- 172.21.15.146:0/957455845 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac009690 con 0x7f68a8004370 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c42aa640 1 -- 172.21.15.146:0/957455845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8004370 msgr2=0x7f68a80048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c42aa640 1 --2- 172.21.15.146:0/957455845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68a8004370 0x7f68a80048c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009ac0 tx=0x7f68a8005830 comp rx=0 tx=0).stop 2024-04-25T15:21:55.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.257+0000 7f68c2847640 1 -- 172.21.15.146:0/957455845 reap_dead start 2024-04-25T15:21:55.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.258+0000 7f68c42aa640 1 -- 172.21.15.146:0/957455845 shutdown_connections 2024-04-25T15:21:55.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.258+0000 7f68c42aa640 1 -- 172.21.15.146:0/957455845 >> 172.21.15.146:0/957455845 conn(0x7f68bc0d2450 msgr2=0x7f68bc0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:55.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.258+0000 7f68c42aa640 1 -- 172.21.15.146:0/957455845 shutdown_connections 2024-04-25T15:21:55.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.258+0000 7f68c42aa640 1 -- 172.21.15.146:0/957455845 wait complete. 2024-04-25T15:21:55.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.259+0000 7f68c42aa640 1 Processor -- start 2024-04-25T15:21:55.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.259+0000 7f68c42aa640 1 -- start start 2024-04-25T15:21:55.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8004370 0x7f68bc15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 0x7f68bc160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68bc1581a0 0x7f68bc160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc0b5150 con 0x7f68bc157d90 2024-04-25T15:21:55.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc06c830 con 0x7f68bc1581a0 2024-04-25T15:21:55.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc0b52d0 con 0x7f68a8004370 2024-04-25T15:21:55.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 0x7f68bc160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 0x7f68bc160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50716/0 (socket says 172.21.15.146:50716) 2024-04-25T15:21:55.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 -- 172.21.15.146:0/4256155462 learned_addr learned my addr 172.21.15.146:0/4256155462 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:55.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c2847640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68bc1581a0 0x7f68bc160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c2046640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8004370 0x7f68bc15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 -- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8004370 msgr2=0x7f68bc15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8004370 0x7f68bc15ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 -- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68bc1581a0 msgr2=0x7f68bc160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68bc1581a0 0x7f68bc160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 -- 172.21.15.146:0/4256155462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc15d430 con 0x7f68bc157d90 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c1845640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 0x7f68bc160680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009be0 tx=0x7f68ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac03f020 con 0x7f68bc157d90 2024-04-25T15:21:55.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ac037040 con 0x7f68bc157d90 2024-04-25T15:21:55.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc15d710 con 0x7f68bc157d90 2024-04-25T15:21:55.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac036070 con 0x7f68bc157d90 2024-04-25T15:21:55.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.261+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc15dcd0 con 0x7f68bc157d90 2024-04-25T15:21:55.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.263+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc06a0e0 con 0x7f68bc157d90 2024-04-25T15:21:55.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.263+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f68ac042020 con 0x7f68bc157d90 2024-04-25T15:21:55.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.263+0000 7f68b2ffd640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68a0074570 0x7f68a0076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.264+0000 7f68c2046640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68a0074570 0x7f68a0076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.264+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f68ac0b5460 con 0x7f68bc157d90 2024-04-25T15:21:55.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.264+0000 7f68c2046640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68a0074570 0x7f68a0076a80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0b4880 tx=0x7f68b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.268+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68ac049030 con 0x7f68bc157d90 2024-04-25T15:21:55.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.446+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68bc0d4100 con 0x7f68bc157d90 2024-04-25T15:21:55.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.447+0000 7f68b2ffd640 1 -- 172.21.15.146:0/4256155462 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7f68ac07f4d0 con 0x7f68bc157d90 2024-04-25T15:21:55.448 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:55.448 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68a0074570 msgr2=0x7f68a0076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68a0074570 0x7f68a0076a80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0b4880 tx=0x7f68b8009040 comp rx=0 tx=0).stop 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 msgr2=0x7f68bc160680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 0x7f68bc160680 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009be0 tx=0x7f68ac004730 comp rx=0 tx=0).stop 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 shutdown_connections 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68a0074570 0x7f68a0076a80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68bc1581a0 0x7f68bc160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68bc157d90 0x7f68bc160680 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 --2- 172.21.15.146:0/4256155462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68a8004370 0x7f68bc15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 >> 172.21.15.146:0/4256155462 conn(0x7f68bc0d2450 msgr2=0x7f68bc15f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 shutdown_connections 2024-04-25T15:21:55.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.450+0000 7f68c42aa640 1 -- 172.21.15.146:0/4256155462 wait complete. 2024-04-25T15:21:55.462 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:55.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:55.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.573+0000 7f6954f38640 1 Processor -- start 2024-04-25T15:21:55.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.574+0000 7f6954f38640 1 -- start start 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f6954f38640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f69500b2cd0 0x7f69500cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f6954f38640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69500cf7d0 con 0x7f69500d2440 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f6954f38640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69500cfab0 con 0x7f69500d1920 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f6954f38640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f695008da90 con 0x7f69500b2cd0 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f694ed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f69500d2440 0x7f6950071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45154/0 (socket says 172.21.15.146:45154) 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f694ed9d640 1 -- 172.21.15.146:0/952244532 learned_addr learned my addr 172.21.15.146:0/952244532 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151499511 0 0) 0x7f69500cf7d0 con 0x7f69500d2440 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.575+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6938003520 con 0x7f69500d2440 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 --2- 172.21.15.146:0/952244532 >> v2:172.21.15.177:3300/0 conn(0x7f69500b2cd0 0x7f69500cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 475251209 0 0) 0x7f6938003520 con 0x7f69500d2440 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 >> v2:172.21.15.177:3300/0 conn(0x7f69500b2cd0 msgr2=0x7f69500cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 --2- 172.21.15.146:0/952244532 >> v2:172.21.15.177:3300/0 conn(0x7f69500b2cd0 0x7f69500cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 >> v1:172.21.15.177:6789/0 conn(0x7f69500d1920 legacy=0x7f69500d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69500cfd90 con 0x7f69500d2440 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f6948001670 con 0x7f69500d2440 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 >> v1:172.21.15.146:6789/0 conn(0x7f69500d2440 legacy=0x7f6950071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f6938003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6938004800 0x7f6938004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6938005250 0x7f693800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f693800c1b0 con 0x7f69380039b0 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f693800c360 con 0x7f6938004800 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f693800c560 con 0x7f6938005250 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f6938003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694dd9b640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6938004800 0x7f6938004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694ed9d640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6938005250 0x7f693800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 -- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6938005250 msgr2=0x7f693800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6938005250 0x7f693800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 -- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6938004800 msgr2=0x7f6938004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6938004800 0x7f6938004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694e59c640 1 -- 172.21.15.146:0/952244532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6938003520 con 0x7f69380039b0 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694ed9d640 1 -- 172.21.15.146:0/952244532 reap_dead start 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.576+0000 7f694dd9b640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6938004800 0x7f6938004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.577+0000 7f694e59c640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f6938003e20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6950075280 tx=0x7f6938005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf31283d6bf4141 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.577+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940009c60 con 0x7f69380039b0 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.577+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6940036070 con 0x7f69380039b0 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.577+0000 7f694d59a640 1 -- 172.21.15.146:0/952244532 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940031810 con 0x7f69380039b0 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 -- 172.21.15.146:0/952244532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 msgr2=0x7f6938003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f6938003e20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6950075280 tx=0x7f6938005830 comp rx=0 tx=0).stop 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 -- 172.21.15.146:0/952244532 shutdown_connections 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 --2- 172.21.15.146:0/952244532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f6938003e20 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 -- 172.21.15.146:0/952244532 >> 172.21.15.146:0/952244532 conn(0x7f69500b52e0 msgr2=0x7f69500b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:55.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 -- 172.21.15.146:0/952244532 shutdown_connections 2024-04-25T15:21:55.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.578+0000 7f6954f38640 1 -- 172.21.15.146:0/952244532 wait complete. 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.580+0000 7f6954f38640 1 Processor -- start 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.580+0000 7f6954f38640 1 -- start start 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f6954f38640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f695006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f6954f38640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6950151e70 0x7f695015fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f6954f38640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69501600e0 0x7f695006def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f6954f38640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f69500cf480 con 0x7f69380039b0 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f6954f38640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69500cf300 con 0x7f69501600e0 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f6954f38640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f69500d0070 con 0x7f6950151e70 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f694e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f695006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f694e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f695006f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50734/0 (socket says 172.21.15.146:50734) 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f694e59c640 1 -- 172.21.15.146:0/1786371642 learned_addr learned my addr 172.21.15.146:0/1786371642 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:55.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f694dd9b640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6950151e70 0x7f695015fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.581+0000 7f694ed9d640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69501600e0 0x7f695006def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694e59c640 1 -- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6950151e70 msgr2=0x7f695015fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694e59c640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6950151e70 0x7f695015fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694e59c640 1 -- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69501600e0 msgr2=0x7f695006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694e59c640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69501600e0 0x7f695006def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694e59c640 1 -- 172.21.15.146:0/1786371642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69501604f0 con 0x7f69380039b0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694dd9b640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6950151e70 0x7f695015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694ed9d640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69501600e0 0x7f695006def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f694e59c640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f695006f020 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f694000e230 tx=0x7f6940009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940048070 con 0x7f69380039b0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6940031d60 con 0x7f69380039b0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695006f560 con 0x7f69380039b0 2024-04-25T15:21:55.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940043650 con 0x7f69380039b0 2024-04-25T15:21:55.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.582+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69501f6690 con 0x7f69380039b0 2024-04-25T15:21:55.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.583+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f695006a0e0 con 0x7f69380039b0 2024-04-25T15:21:55.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.584+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f694004b070 con 0x7f69380039b0 2024-04-25T15:21:55.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.584+0000 7f693effd640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6930074540 0x7f6930076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.584+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f69400b5c70 con 0x7f69380039b0 2024-04-25T15:21:55.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.584+0000 7f694dd9b640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6930074540 0x7f6930076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.585+0000 7f694dd9b640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6930074540 0x7f6930076a50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6944000c00 tx=0x7f6944009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.589+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69501604f0 con 0x7f69380039b0 2024-04-25T15:21:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.762+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f69500b8c30 con 0x7f69380039b0 2024-04-25T15:21:55.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.763+0000 7f693effd640 1 -- 172.21.15.146:0/1786371642 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f694002e050 con 0x7f69380039b0 2024-04-25T15:21:55.764 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:55.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6930074540 msgr2=0x7f6930076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6930074540 0x7f6930076a50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6944000c00 tx=0x7f6944009040 comp rx=0 tx=0).stop 2024-04-25T15:21:55.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 msgr2=0x7f695006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f695006f020 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f694000e230 tx=0x7f6940009750 comp rx=0 tx=0).stop 2024-04-25T15:21:55.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 shutdown_connections 2024-04-25T15:21:55.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6930074540 0x7f6930076a50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69501600e0 0x7f695006def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6950151e70 0x7f695015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 --2- 172.21.15.146:0/1786371642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69380039b0 0x7f695006f020 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 >> 172.21.15.146:0/1786371642 conn(0x7f69500b52e0 msgr2=0x7f69500b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 shutdown_connections 2024-04-25T15:21:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.765+0000 7f6954f38640 1 -- 172.21.15.146:0/1786371642 wait complete. 2024-04-25T15:21:55.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:55.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.888+0000 7ff7a524c640 1 Processor -- start 2024-04-25T15:21:55.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.888+0000 7ff7a524c640 1 -- start start 2024-04-25T15:21:55.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff7a524c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00bb450 0x7ff7a00d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff7a524c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff7a00d0de0 0x7ff7a00b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff7a524c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a00b5230 con 0x7ff7a0163c00 2024-04-25T15:21:55.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff7a524c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff7a006c8c0 con 0x7ff7a00d0de0 2024-04-25T15:21:55.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff7a524c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a006cba0 con 0x7ff7a00bb450 2024-04-25T15:21:55.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff79f59e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff7a00d0de0 0x7ff7a00b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff79ed9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff7a0163c00 0x7ff7a006d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:39820/0 (socket says 172.21.15.146:39820) 2024-04-25T15:21:55.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff79e59c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00bb450 0x7ff7a00d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.889+0000 7ff79ed9d640 1 -- 172.21.15.146:0/668372235 learned_addr learned my addr 172.21.15.146:0/668372235 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:55.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79dd9b640 1 -- 172.21.15.146:0/668372235 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021669233 0 0) 0x7ff7a00b5230 con 0x7ff7a0163c00 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79dd9b640 1 -- 172.21.15.146:0/668372235 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff788003520 con 0x7ff7a0163c00 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79e59c640 1 -- 172.21.15.146:0/668372235 >> v2:172.21.15.177:3300/0 conn(0x7ff7a00d0de0 msgr2=0x7ff7a00b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79e59c640 1 --2- 172.21.15.146:0/668372235 >> v2:172.21.15.177:3300/0 conn(0x7ff7a00d0de0 0x7ff7a00b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79e59c640 1 -- 172.21.15.146:0/668372235 >> v1:172.21.15.177:6789/0 conn(0x7ff7a0163c00 legacy=0x7ff7a006d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79e59c640 1 -- 172.21.15.146:0/668372235 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a006ce80 con 0x7ff7a00bb450 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79f59e640 1 --2- 172.21.15.146:0/668372235 >> v2:172.21.15.177:3300/0 conn(0x7ff7a00d0de0 0x7ff7a00b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.890+0000 7ff79e59c640 1 --2- 172.21.15.146:0/668372235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00bb450 0x7ff7a00d08a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7940097a0 tx=0x7ff79402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ce6a892e32f685 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff79dd9b640 1 -- 172.21.15.146:0/668372235 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794037070 con 0x7ff7a00bb450 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff79dd9b640 1 -- 172.21.15.146:0/668372235 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79402ae50 con 0x7ff7a00bb450 2024-04-25T15:21:55.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff79dd9b640 1 -- 172.21.15.146:0/668372235 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794032640 con 0x7ff7a00bb450 2024-04-25T15:21:55.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 -- 172.21.15.146:0/668372235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00bb450 msgr2=0x7ff7a00d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 --2- 172.21.15.146:0/668372235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00bb450 0x7ff7a00d08a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7940097a0 tx=0x7ff79402a100 comp rx=0 tx=0).stop 2024-04-25T15:21:55.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 -- 172.21.15.146:0/668372235 shutdown_connections 2024-04-25T15:21:55.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 --2- 172.21.15.146:0/668372235 >> v2:172.21.15.177:3300/0 conn(0x7ff7a00d0de0 0x7ff7a00b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 --2- 172.21.15.146:0/668372235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00bb450 0x7ff7a00d08a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 -- 172.21.15.146:0/668372235 >> 172.21.15.146:0/668372235 conn(0x7ff7a00b7970 msgr2=0x7ff7a00b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:55.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 -- 172.21.15.146:0/668372235 shutdown_connections 2024-04-25T15:21:55.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.891+0000 7ff7a524c640 1 -- 172.21.15.146:0/668372235 wait complete. 2024-04-25T15:21:55.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.893+0000 7ff7a524c640 1 Processor -- start 2024-04-25T15:21:55.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.893+0000 7ff7a524c640 1 -- start start 2024-04-25T15:21:55.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7a524c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 0x7ff7a01f9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7a524c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00d0de0 0x7ff7a01fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7a524c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7a0163c00 0x7ff7a01e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7a524c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a00bb860 con 0x7ff7a00bb450 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7a524c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a006c550 con 0x7ff7a0163c00 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7a524c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a00bb9e0 con 0x7ff7a00d0de0 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 0x7ff7a01f9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 0x7ff7a01f9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50742/0 (socket says 172.21.15.146:50742) 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 -- 172.21.15.146:0/1441347911 learned_addr learned my addr 172.21.15.146:0/1441347911 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79f59e640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7a0163c00 0x7ff7a01e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79e59c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00d0de0 0x7ff7a01fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 -- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00d0de0 msgr2=0x7ff7a01fa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00d0de0 0x7ff7a01fa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 -- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7a0163c00 msgr2=0x7ff7a01e9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7a0163c00 0x7ff7a01e9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 -- 172.21.15.146:0/1441347911 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a01faa30 con 0x7ff7a00bb450 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff79ed9d640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 0x7ff7a01f9c90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c390 tx=0x7ff79000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.895+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790017070 con 0x7ff7a00bb450 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.896+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff790005ce0 con 0x7ff7a00bb450 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.896+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a00cf0d0 con 0x7ff7a00bb450 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.896+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7900072f0 con 0x7ff7a00bb450 2024-04-25T15:21:55.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.896+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a00cf690 con 0x7ff7a00bb450 2024-04-25T15:21:55.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.897+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a006a0e0 con 0x7ff7a00bb450 2024-04-25T15:21:55.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.897+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7ff79001a070 con 0x7ff7a00bb450 2024-04-25T15:21:55.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.898+0000 7ff7877fe640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff78c074540 0x7ff78c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:55.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.898+0000 7ff79e59c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff78c074540 0x7ff78c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:55.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.898+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7ff790094680 con 0x7ff7a00bb450 2024-04-25T15:21:55.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.898+0000 7ff79e59c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff78c074540 0x7ff78c076a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff794034000 tx=0x7ff79402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:55.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:55.903+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff79005e660 con 0x7ff7a00bb450 2024-04-25T15:21:56.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.080+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff7a00b4e80 con 0x7ff7a00bb450 2024-04-25T15:21:56.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.080+0000 7ff7877fe640 1 -- 172.21.15.146:0/1441347911 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7ff790062960 con 0x7ff7a00bb450 2024-04-25T15:21:56.082 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:56.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff78c074540 msgr2=0x7ff78c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff78c074540 0x7ff78c076a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff794034000 tx=0x7ff79402e000 comp rx=0 tx=0).stop 2024-04-25T15:21:56.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 msgr2=0x7ff7a01f9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 0x7ff7a01f9c90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c390 tx=0x7ff79000c790 comp rx=0 tx=0).stop 2024-04-25T15:21:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 shutdown_connections 2024-04-25T15:21:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff78c074540 0x7ff78c076a50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7a0163c00 0x7ff7a01e9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff7a00d0de0 0x7ff7a01fa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 --2- 172.21.15.146:0/1441347911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00bb450 0x7ff7a01f9c90 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 >> 172.21.15.146:0/1441347911 conn(0x7ff7a00b7970 msgr2=0x7ff7a0162c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:56.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 shutdown_connections 2024-04-25T15:21:56.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.082+0000 7ff7a524c640 1 -- 172.21.15.146:0/1441347911 wait complete. 2024-04-25T15:21:56.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:56.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.205+0000 7fba6fbf3640 1 Processor -- start 2024-04-25T15:21:56.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.206+0000 7fba6fbf3640 1 -- start start 2024-04-25T15:21:56.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6fbf3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba680ccd40 0x7fba6809bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6fbf3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba680d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6fbf3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba680d2fd0 con 0x7fba680b2400 2024-04-25T15:21:56.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6fbf3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fba680b1960 con 0x7fba680ccd40 2024-04-25T15:21:56.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6fbf3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fba680b3ce0 con 0x7fba680d2680 2024-04-25T15:21:56.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6d98f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba680ccd40 0x7fba6809bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6e190640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fba680b2400 0x7fba680b2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45164/0 (socket says 172.21.15.146:45164) 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6e190640 1 -- 172.21.15.146:0/1389004356 learned_addr learned my addr 172.21.15.146:0/1389004356 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6d18e640 1 --2- 172.21.15.146:0/1389004356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba680d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6d98f640 1 -- 172.21.15.146:0/1389004356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 msgr2=0x7fba680d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6d98f640 1 --2- 172.21.15.146:0/1389004356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba680d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6d98f640 1 -- 172.21.15.146:0/1389004356 >> v1:172.21.15.146:6789/0 conn(0x7fba680b2400 legacy=0x7fba680b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.208+0000 7fba6d98f640 1 -- 172.21.15.146:0/1389004356 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba680b1c00 con 0x7fba680ccd40 2024-04-25T15:21:56.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.207+0000 7fba6c98d640 1 -- 172.21.15.146:0/1389004356 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564024903 0 0) 0x7fba60001110 con 0x7fba680b2400 2024-04-25T15:21:56.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.208+0000 7fba6d98f640 1 --2- 172.21.15.146:0/1389004356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6809bba0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fba64009de0 tx=0x7fba6402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2fc3ddf50f3e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:56.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.208+0000 7fba6c98d640 1 -- 172.21.15.146:0/1389004356 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64007740 con 0x7fba680ccd40 2024-04-25T15:21:56.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.208+0000 7fba6c98d640 1 -- 172.21.15.146:0/1389004356 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba640078e0 con 0x7fba680ccd40 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.208+0000 7fba6c98d640 1 -- 172.21.15.146:0/1389004356 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64007cb0 con 0x7fba680ccd40 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 -- 172.21.15.146:0/1389004356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 msgr2=0x7fba6809bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/1389004356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6809bba0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fba64009de0 tx=0x7fba6402a6c0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 -- 172.21.15.146:0/1389004356 shutdown_connections 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/1389004356 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba680d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/1389004356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6809bba0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 -- 172.21.15.146:0/1389004356 >> 172.21.15.146:0/1389004356 conn(0x7fba680b7400 msgr2=0x7fba680b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 -- 172.21.15.146:0/1389004356 shutdown_connections 2024-04-25T15:21:56.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.209+0000 7fba6fbf3640 1 -- 172.21.15.146:0/1389004356 wait complete. 2024-04-25T15:21:56.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.211+0000 7fba6fbf3640 1 Processor -- start 2024-04-25T15:21:56.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.211+0000 7fba6fbf3640 1 -- start start 2024-04-25T15:21:56.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6fbf3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba680b2400 0x7fba680cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6fbf3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6fbf3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba681f9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6fbf3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fba680b3940 con 0x7fba680ccd40 2024-04-25T15:21:56.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6fbf3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba680b3fc0 con 0x7fba680b2400 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6fbf3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fba680b3ac0 con 0x7fba680d2680 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6d18e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6d18e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6815efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50768/0 (socket says 172.21.15.146:50768) 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.212+0000 7fba6d18e640 1 -- 172.21.15.146:0/3557193662 learned_addr learned my addr 172.21.15.146:0/3557193662 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d98f640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba680b2400 0x7fba680cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6e190640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba681f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d18e640 1 -- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 msgr2=0x7fba681f9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d18e640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba681f9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d18e640 1 -- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba680b2400 msgr2=0x7fba680cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d18e640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba680b2400 0x7fba680cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d18e640 1 -- 172.21.15.146:0/3557193662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba680cb0a0 con 0x7fba680ccd40 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d98f640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba680b2400 0x7fba680cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6e190640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba681f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:56.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6d18e640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6815efe0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fba5800e8a0 tx=0x7fba5800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58017070 con 0x7fba680ccd40 2024-04-25T15:21:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5800c940 con 0x7fba680ccd40 2024-04-25T15:21:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba680ca300 con 0x7fba680ccd40 2024-04-25T15:21:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5800cd30 con 0x7fba680ccd40 2024-04-25T15:21:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.213+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba680ca5c0 con 0x7fba680ccd40 2024-04-25T15:21:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.214+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba680cc4f0 con 0x7fba680ccd40 2024-04-25T15:21:56.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.215+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fba5801a070 con 0x7fba680ccd40 2024-04-25T15:21:56.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.215+0000 7fba5e7fc640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fba4c074570 0x7fba4c076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.215+0000 7fba6d98f640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fba4c074570 0x7fba4c076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.215+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fba58095230 con 0x7fba680ccd40 2024-04-25T15:21:56.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.216+0000 7fba6d98f640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fba4c074570 0x7fba4c076a80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fba64034000 tx=0x7fba64033040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.220+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba5805f8f0 con 0x7fba680ccd40 2024-04-25T15:21:56.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.400+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fba6806a0e0 con 0x7fba680ccd40 2024-04-25T15:21:56.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.400+0000 7fba5e7fc640 1 -- 172.21.15.146:0/3557193662 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fba5805f1f0 con 0x7fba680ccd40 2024-04-25T15:21:56.401 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fba4c074570 msgr2=0x7fba4c076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fba4c074570 0x7fba4c076a80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fba64034000 tx=0x7fba64033040 comp rx=0 tx=0).stop 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 msgr2=0x7fba6815efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6815efe0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fba5800e8a0 tx=0x7fba5800ed60 comp rx=0 tx=0).stop 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 shutdown_connections 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fba4c074570 0x7fba4c076a80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680d2680 0x7fba681f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ccd40 0x7fba6815efe0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 --2- 172.21.15.146:0/3557193662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba680b2400 0x7fba680cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 >> 172.21.15.146:0/3557193662 conn(0x7fba680b7400 msgr2=0x7fba680af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 shutdown_connections 2024-04-25T15:21:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.402+0000 7fba6fbf3640 1 -- 172.21.15.146:0/3557193662 wait complete. 2024-04-25T15:21:56.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:21:56.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.524+0000 7f0110dce640 1 Processor -- start 2024-04-25T15:21:56.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.524+0000 7f0110dce640 1 -- start start 2024-04-25T15:21:56.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f0110dce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f010c0cfbe0 0x7f010c0cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f0110dce640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f010c0cce10 0x7f010c0cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f0110dce640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0b4a60 con 0x7f010c0d19d0 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f0110dce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f010c0b5020 con 0x7f010c0cfbe0 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f0110dce640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f010c0b4d40 con 0x7f010c0cce10 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f010c0cfbe0 0x7f010c0cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f010c0cfbe0 0x7f010c0cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50776/0 (socket says 172.21.15.146:50776) 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 -- 172.21.15.146:0/1581843062 learned_addr learned my addr 172.21.15.146:0/1581843062 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010ad9d640 1 --2- 172.21.15.146:0/1581843062 >> v2:172.21.15.177:3300/0 conn(0x7f010c0cce10 0x7f010c0cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 -- 172.21.15.146:0/1581843062 >> v2:172.21.15.177:3300/0 conn(0x7f010c0cce10 msgr2=0x7f010c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 --2- 172.21.15.146:0/1581843062 >> v2:172.21.15.177:3300/0 conn(0x7f010c0cce10 0x7f010c0cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 -- 172.21.15.146:0/1581843062 >> v1:172.21.15.177:6789/0 conn(0x7f010c0d19d0 legacy=0x7f010c0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.526+0000 7f010a59c640 1 -- 172.21.15.146:0/1581843062 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c0cead0 con 0x7f010c0cfbe0 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.527+0000 7f010a59c640 1 --2- 172.21.15.146:0/1581843062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c0cf260 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f010c0d0210 tx=0x7f00fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aeb685f1e772280 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:56.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.527+0000 7f010959a640 1 -- 172.21.15.146:0/1581843062 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc004030 con 0x7f010c0cfbe0 2024-04-25T15:21:56.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.527+0000 7f010959a640 1 -- 172.21.15.146:0/1581843062 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00fc0041d0 con 0x7f010c0cfbe0 2024-04-25T15:21:56.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.527+0000 7f010959a640 1 -- 172.21.15.146:0/1581843062 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc003610 con 0x7f010c0cfbe0 2024-04-25T15:21:56.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 -- 172.21.15.146:0/1581843062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 msgr2=0x7f010c0cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 --2- 172.21.15.146:0/1581843062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c0cf260 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f010c0d0210 tx=0x7f00fc02a270 comp rx=0 tx=0).stop 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 -- 172.21.15.146:0/1581843062 shutdown_connections 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 --2- 172.21.15.146:0/1581843062 >> v2:172.21.15.177:3300/0 conn(0x7f010c0cce10 0x7f010c0cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 --2- 172.21.15.146:0/1581843062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c0cf260 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 -- 172.21.15.146:0/1581843062 >> 172.21.15.146:0/1581843062 conn(0x7f010c0b73b0 msgr2=0x7f010c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 -- 172.21.15.146:0/1581843062 shutdown_connections 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.528+0000 7f0110dce640 1 -- 172.21.15.146:0/1581843062 wait complete. 2024-04-25T15:21:56.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.530+0000 7f0110dce640 1 Processor -- start 2024-04-25T15:21:56.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.530+0000 7f0110dce640 1 -- start start 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0110dce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f010c0cce10 0x7f010c1599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0110dce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0110dce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f010c0d19d0 0x7f010c0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0110dce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f010c0cf7a0 con 0x7f010c0cfbe0 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0110dce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f010c0cd760 con 0x7f010c0cce10 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0110dce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f010c15a4b0 con 0x7f010c0d19d0 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0109d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0109d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c14fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50788/0 (socket says 172.21.15.146:50788) 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f0109d9b640 1 -- 172.21.15.146:0/4275330685 learned_addr learned my addr 172.21.15.146:0/4275330685 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f010a59c640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f010c0cce10 0x7f010c1599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.531+0000 7f010ad9d640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f010c0d19d0 0x7f010c0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0109d9b640 1 -- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f010c0d19d0 msgr2=0x7f010c0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0109d9b640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f010c0d19d0 0x7f010c0ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0109d9b640 1 -- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f010c0cce10 msgr2=0x7f010c1599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0109d9b640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f010c0cce10 0x7f010c1599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0109d9b640 1 -- 172.21.15.146:0/4275330685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c1f6680 con 0x7f010c0cfbe0 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f010ad9d640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f010c0d19d0 0x7f010c0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0109d9b640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c14fa10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f010000c470 tx=0x7f010000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01000052d0 con 0x7f010c0cfbe0 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0100005490 con 0x7f010c0cfbe0 2024-04-25T15:21:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010c0cbf50 con 0x7f010c0cfbe0 2024-04-25T15:21:56.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100007260 con 0x7f010c0cfbe0 2024-04-25T15:21:56.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.532+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010c0c9980 con 0x7f010c0cfbe0 2024-04-25T15:21:56.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.533+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c06a0e0 con 0x7f010c0cfbe0 2024-04-25T15:21:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.533+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f0100007400 con 0x7f010c0cfbe0 2024-04-25T15:21:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.534+0000 7f00faffd640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00ec074540 0x7f00ec076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.534+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0100094830 con 0x7f010c0cfbe0 2024-04-25T15:21:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.534+0000 7f010a59c640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00ec074540 0x7f00ec076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.535+0000 7f010a59c640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00ec074540 0x7f00ec076a50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f00fc033000 tx=0x7f00fc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:56.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.539+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f010c1f6680 con 0x7f010c0cfbe0 2024-04-25T15:21:56.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.727+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f010c0cf920 con 0x7f010c0cfbe0 2024-04-25T15:21:56.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.728+0000 7f00faffd640 1 -- 172.21.15.146:0/4275330685 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f010005e780 con 0x7f010c0cfbe0 2024-04-25T15:21:56.729 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6835/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:21:56.729 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6833/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:21:56.729 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00ec074540 msgr2=0x7f00ec076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00ec074540 0x7f00ec076a50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f00fc033000 tx=0x7f00fc02e000 comp rx=0 tx=0).stop 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 msgr2=0x7f010c14fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c14fa10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f010000c470 tx=0x7f010000c930 comp rx=0 tx=0).stop 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 shutdown_connections 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00ec074540 0x7f00ec076a50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f010c0d19d0 0x7f010c0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010c0cfbe0 0x7f010c14fa10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 --2- 172.21.15.146:0/4275330685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f010c0cce10 0x7f010c1599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:56.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 >> 172.21.15.146:0/4275330685 conn(0x7f010c0b73b0 msgr2=0x7f010c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:56.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 shutdown_connections 2024-04-25T15:21:56.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:56.730+0000 7f0110dce640 1 -- 172.21.15.146:0/4275330685 wait complete. 2024-04-25T15:21:56.742 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:21:56.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:56.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T15:21:56.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[53345]: fuse finished with error 0 and tester_r 0 2024-04-25T15:21:56.766 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:21:56.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:56.766 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:21:56.934 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:21:56.934 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:21:56.934 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:56.952 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:21:56.953 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:57.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:57.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.118+0000 7ff893301640 1 Processor -- start 2024-04-25T15:21:57.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.119+0000 7ff893301640 1 -- start start 2024-04-25T15:21:57.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff893301640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff88c0715e0 0x7ff88c06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff893301640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c070190 0x7ff88c169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff893301640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88c0705a0 con 0x7ff88c0b7350 2024-04-25T15:21:57.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff893301640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c071bc0 con 0x7ff88c0715e0 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff893301640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c0d1b00 con 0x7ff88c070190 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff89089c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff88c0715e0 0x7ff88c06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff89189e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c070190 0x7ff88c169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff89109d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7ff88c0b7350 0x7ff88c160000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:39838/0 (socket says 172.21.15.146:39838) 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.120+0000 7ff89109d640 1 -- 172.21.15.146:0/3809884877 learned_addr learned my addr 172.21.15.146:0/3809884877 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff89189e640 1 -- 172.21.15.146:0/3809884877 >> v2:172.21.15.177:3300/0 conn(0x7ff88c0715e0 msgr2=0x7ff88c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff89189e640 1 --2- 172.21.15.146:0/3809884877 >> v2:172.21.15.177:3300/0 conn(0x7ff88c0715e0 0x7ff88c06fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff89189e640 1 -- 172.21.15.146:0/3809884877 >> v1:172.21.15.177:6789/0 conn(0x7ff88c0b7350 legacy=0x7ff88c160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff883fff640 1 -- 172.21.15.146:0/3809884877 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609538068 0 0) 0x7ff888001110 con 0x7ff88c0b7350 2024-04-25T15:21:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff89189e640 1 -- 172.21.15.146:0/3809884877 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c0d1de0 con 0x7ff88c070190 2024-04-25T15:21:57.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff89089c640 1 --2- 172.21.15.146:0/3809884877 >> v2:172.21.15.177:3300/0 conn(0x7ff88c0715e0 0x7ff88c06fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:57.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff89189e640 1 --2- 172.21.15.146:0/3809884877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c070190 0x7ff88c169ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff884009d90 tx=0x7ff88402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6a38eb2fd50a7de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff883fff640 1 -- 172.21.15.146:0/3809884877 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884007740 con 0x7ff88c070190 2024-04-25T15:21:57.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.121+0000 7ff883fff640 1 -- 172.21.15.146:0/3809884877 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8840078e0 con 0x7ff88c070190 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff883fff640 1 -- 172.21.15.146:0/3809884877 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884007cb0 con 0x7ff88c070190 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 -- 172.21.15.146:0/3809884877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c070190 msgr2=0x7ff88c169ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 --2- 172.21.15.146:0/3809884877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c070190 0x7ff88c169ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff884009d90 tx=0x7ff88402a660 comp rx=0 tx=0).stop 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 -- 172.21.15.146:0/3809884877 shutdown_connections 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 --2- 172.21.15.146:0/3809884877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c070190 0x7ff88c169ca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 --2- 172.21.15.146:0/3809884877 >> v2:172.21.15.177:3300/0 conn(0x7ff88c0715e0 0x7ff88c06fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 -- 172.21.15.146:0/3809884877 >> 172.21.15.146:0/3809884877 conn(0x7ff88c09b2b0 msgr2=0x7ff88c0b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 -- 172.21.15.146:0/3809884877 shutdown_connections 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.122+0000 7ff893301640 1 -- 172.21.15.146:0/3809884877 wait complete. 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.124+0000 7ff893301640 1 Processor -- start 2024-04-25T15:21:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.124+0000 7ff893301640 1 -- start start 2024-04-25T15:21:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff893301640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 0x7ff88c1501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff893301640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff88c0715e0 0x7ff88c1e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff893301640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c0b7350 0x7ff88c1e5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff893301640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c0cf4b0 con 0x7ff88c070190 2024-04-25T15:21:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff893301640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c0d23a0 con 0x7ff88c0715e0 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff893301640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c070980 con 0x7ff88c0b7350 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff89109d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 0x7ff88c1501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff89109d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 0x7ff88c1501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50794/0 (socket says 172.21.15.146:50794) 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff89109d640 1 -- 172.21.15.146:0/1979731649 learned_addr learned my addr 172.21.15.146:0/1979731649 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89089c640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff88c0715e0 0x7ff88c1e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.125+0000 7ff89189e640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c0b7350 0x7ff88c1e5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89109d640 1 -- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c0b7350 msgr2=0x7ff88c1e5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89109d640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c0b7350 0x7ff88c1e5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89109d640 1 -- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff88c0715e0 msgr2=0x7ff88c1e4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89109d640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff88c0715e0 0x7ff88c1e4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89109d640 1 -- 172.21.15.146:0/1979731649 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c1506e0 con 0x7ff88c070190 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff89109d640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 0x7ff88c1501a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c410 tx=0x7ff88800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8880074e0 con 0x7ff88c070190 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8880047e0 con 0x7ff88c070190 2024-04-25T15:21:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88c1ea660 con 0x7ff88c070190 2024-04-25T15:21:57.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888004bd0 con 0x7ff88c070190 2024-04-25T15:21:57.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.126+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88c1eac20 con 0x7ff88c070190 2024-04-25T15:21:57.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.127+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88c06a0e0 con 0x7ff88c070190 2024-04-25T15:21:57.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.128+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7ff888011070 con 0x7ff88c070190 2024-04-25T15:21:57.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.128+0000 7ff881ffb640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff870074540 0x7ff870076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.128+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7ff888094fc0 con 0x7ff88c070190 2024-04-25T15:21:57.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.128+0000 7ff89089c640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff870074540 0x7ff870076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.129+0000 7ff89089c640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff870074540 0x7ff870076a50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff88c15eef0 tx=0x7ff87c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.133+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff88c0705a0 con 0x7ff88c070190 2024-04-25T15:21:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.310+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff88c0711b0 con 0x7ff88c070190 2024-04-25T15:21:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.311+0000 7ff881ffb640 1 -- 172.21.15.146:0/1979731649 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7ff88805ef10 con 0x7ff88c070190 2024-04-25T15:21:57.312 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:57.312 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff870074540 msgr2=0x7ff870076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff870074540 0x7ff870076a50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff88c15eef0 tx=0x7ff87c009190 comp rx=0 tx=0).stop 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 msgr2=0x7ff88c1501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 0x7ff88c1501a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c410 tx=0x7ff88800c8d0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 shutdown_connections 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff870074540 0x7ff870076a50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff88c0b7350 0x7ff88c1e5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff88c0715e0 0x7ff88c1e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 --2- 172.21.15.146:0/1979731649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c070190 0x7ff88c1501a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 >> 172.21.15.146:0/1979731649 conn(0x7ff88c09b2b0 msgr2=0x7ff88c06c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 shutdown_connections 2024-04-25T15:21:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.313+0000 7ff893301640 1 -- 172.21.15.146:0/1979731649 wait complete. 2024-04-25T15:21:57.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:57.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:57.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.436+0000 7fca92d64640 1 Processor -- start 2024-04-25T15:21:57.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.436+0000 7fca92d64640 1 -- start start 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.437+0000 7fca92d64640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca8c0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.437+0000 7fca92d64640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8c08d590 con 0x7fca8c0b4ca0 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.437+0000 7fca92d64640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8c0d27d0 con 0x7fca8c08dd50 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.437+0000 7fca92d64640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca8c0b9340 con 0x7fca8c0b8f30 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.437+0000 7fca8bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fca8c0b4ca0 0x7fca8c0ce6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45170/0 (socket says 172.21.15.146:45170) 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.437+0000 7fca8bfff640 1 -- 172.21.15.146:0/1622914439 learned_addr learned my addr 172.21.15.146:0/1622914439 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2816229500 0 0) 0x7fca8c08d590 con 0x7fca8c0b4ca0 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca78003520 con 0x7fca8c0b4ca0 2024-04-25T15:21:57.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca91301640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca8c0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 410675285 0 0) 0x7fca78003520 con 0x7fca8c0b4ca0 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 msgr2=0x7fca8c0cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca8c0cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 >> v1:172.21.15.177:6789/0 conn(0x7fca8c08dd50 legacy=0x7fca8c0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8c0b48b0 con 0x7fca8c0b4ca0 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fca7c001a10 con 0x7fca8c0b4ca0 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca91301640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca8c0cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 >> v1:172.21.15.146:6789/0 conn(0x7fca8c0b4ca0 legacy=0x7fca8c0ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca780039e0 0x7fca78003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78004d40 0x7fca780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c0b9340 con 0x7fca780039e0 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca7800bb80 con 0x7fca78004d40 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca7800bd80 con 0x7fca8c0b8f30 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca90b00640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca780039e0 0x7fca78003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca91301640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78004d40 0x7fca780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca8bfff640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca90b00640 1 -- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 msgr2=0x7fca78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca90b00640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c0b8f30 0x7fca78004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca90b00640 1 -- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78004d40 msgr2=0x7fca780051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca90b00640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca78004d40 0x7fca780051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.438+0000 7fca90b00640 1 -- 172.21.15.146:0/1622914439 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca78003520 con 0x7fca780039e0 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.439+0000 7fca90b00640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca780039e0 0x7fca78003e50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fca74001720 tx=0x7fca78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e39fcb42915b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.439+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca74001370 con 0x7fca780039e0 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.439+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca740052d0 con 0x7fca780039e0 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.439+0000 7fca8b7fe640 1 -- 172.21.15.146:0/1622914439 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7402abc0 con 0x7fca780039e0 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca92d64640 1 -- 172.21.15.146:0/1622914439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca780039e0 msgr2=0x7fca78003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca92d64640 1 --2- 172.21.15.146:0/1622914439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca780039e0 0x7fca78003e50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fca74001720 tx=0x7fca78005830 comp rx=0 tx=0).stop 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca91301640 1 -- 172.21.15.146:0/1622914439 reap_dead start 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca92d64640 1 -- 172.21.15.146:0/1622914439 shutdown_connections 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca92d64640 1 -- 172.21.15.146:0/1622914439 >> 172.21.15.146:0/1622914439 conn(0x7fca8c0b5430 msgr2=0x7fca8c0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca92d64640 1 -- 172.21.15.146:0/1622914439 shutdown_connections 2024-04-25T15:21:57.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.440+0000 7fca92d64640 1 -- 172.21.15.146:0/1622914439 wait complete. 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.442+0000 7fca92d64640 1 Processor -- start 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.442+0000 7fca92d64640 1 -- start start 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca92d64640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca780039e0 0x7fca8c1dc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca92d64640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c1d59a0 0x7fca8c1e8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca92d64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 0x7fca8c1e8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca92d64640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c0d2340 con 0x7fca8c1d5db0 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca92d64640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c08d8c0 con 0x7fca780039e0 2024-04-25T15:21:57.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca92d64640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fca8c0d24c0 con 0x7fca8c1d59a0 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca91301640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 0x7fca8c1e8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca91301640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 0x7fca8c1e8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50808/0 (socket says 172.21.15.146:50808) 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca91301640 1 -- 172.21.15.146:0/138203365 learned_addr learned my addr 172.21.15.146:0/138203365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca8bfff640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c1d59a0 0x7fca8c1e8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.443+0000 7fca90b00640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca780039e0 0x7fca8c1dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca91301640 1 -- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c1d59a0 msgr2=0x7fca8c1e8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca91301640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c1d59a0 0x7fca8c1e8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca91301640 1 -- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca780039e0 msgr2=0x7fca8c1dc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca91301640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca780039e0 0x7fca8c1dc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca91301640 1 -- 172.21.15.146:0/138203365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8c1dc7a0 con 0x7fca8c1d5db0 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca8bfff640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c1d59a0 0x7fca8c1e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca90b00640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca780039e0 0x7fca8c1dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:57.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca91301640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 0x7fca8c1e8c50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fca84006fd0 tx=0x7fca84010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8400f040 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8401c070 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca8c1d9ac0 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84003610 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.444+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8c1da080 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.445+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca8c0d1c30 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.446+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fca840037b0 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.446+0000 7fca897fa640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca6c074570 0x7fca6c076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.446+0000 7fca90b00640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca6c074570 0x7fca6c076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.446+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fca84095270 con 0x7fca8c1d5db0 2024-04-25T15:21:57.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.447+0000 7fca90b00640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca6c074570 0x7fca6c076a80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca74032000 tx=0x7fca7402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.451+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca8c0d1c30 con 0x7fca8c1d5db0 2024-04-25T15:21:57.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.626+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fca8c068c30 con 0x7fca8c1d5db0 2024-04-25T15:21:57.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.626+0000 7fca897fa640 1 -- 172.21.15.146:0/138203365 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fca8405f710 con 0x7fca8c1d5db0 2024-04-25T15:21:57.627 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca6c074570 msgr2=0x7fca6c076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca6c074570 0x7fca6c076a80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca74032000 tx=0x7fca7402e000 comp rx=0 tx=0).stop 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 msgr2=0x7fca8c1e8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 0x7fca8c1e8c50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fca84006fd0 tx=0x7fca84010930 comp rx=0 tx=0).stop 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 shutdown_connections 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fca6c074570 0x7fca6c076a80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca8c1d5db0 0x7fca8c1e8c50 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca8c1d59a0 0x7fca8c1e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 --2- 172.21.15.146:0/138203365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca780039e0 0x7fca8c1dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 >> 172.21.15.146:0/138203365 conn(0x7fca8c0b5430 msgr2=0x7fca8c072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:57.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 shutdown_connections 2024-04-25T15:21:57.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.628+0000 7fca92d64640 1 -- 172.21.15.146:0/138203365 wait complete. 2024-04-25T15:21:57.640 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:57.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.753+0000 7fa59a338640 1 Processor -- start 2024-04-25T15:21:57.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.753+0000 7fa59a338640 1 -- start start 2024-04-25T15:21:57.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.754+0000 7fa59a338640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa5940d2e90 0x7fa594165a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa59a338640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5940b2d90 con 0x7fa5940b3940 2024-04-25T15:21:57.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa59a338640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5940b3050 con 0x7fa5940b3d50 2024-04-25T15:21:57.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa59a338640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa5940b2b60 con 0x7fa5940d2e90 2024-04-25T15:21:57.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa593fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa5940b3940 0x7fa5940d2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45178/0 (socket says 172.21.15.146:45178) 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa593fff640 1 -- 172.21.15.146:0/1754514910 learned_addr learned my addr 172.21.15.146:0/1754514910 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585748684 0 0) 0x7fa5940b2d90 con 0x7fa5940b3940 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa57c003520 con 0x7fa5940b3940 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa5937fe640 1 --2- 172.21.15.146:0/1754514910 >> v2:172.21.15.177:3300/0 conn(0x7fa5940d2e90 0x7fa594165a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3280050515 0 0) 0x7fa57c003520 con 0x7fa5940b3940 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 >> v2:172.21.15.177:3300/0 conn(0x7fa5940d2e90 msgr2=0x7fa594165a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 --2- 172.21.15.146:0/1754514910 >> v2:172.21.15.177:3300/0 conn(0x7fa5940d2e90 0x7fa594165a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 >> v1:172.21.15.177:6789/0 conn(0x7fa5940b3d50 legacy=0x7fa594167fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5940b3330 con 0x7fa5940b3940 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fa584001670 con 0x7fa5940b3940 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 >> v1:172.21.15.146:6789/0 conn(0x7fa5940b3940 legacy=0x7fa5940d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa57c0039b0 0x7fa57c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c004800 0x7fa57c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa57c005300 0x7fa57c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c00c270 con 0x7fa57c0039b0 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa57c00c420 con 0x7fa57c004800 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa57c00c620 con 0x7fa57c005300 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa593fff640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa57c0039b0 0x7fa57c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.755+0000 7fa5937fe640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c004800 0x7fa57c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa5988d5640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa57c005300 0x7fa57c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa593fff640 1 -- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa57c005300 msgr2=0x7fa57c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa593fff640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa57c005300 0x7fa57c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa593fff640 1 -- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c004800 msgr2=0x7fa57c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa593fff640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c004800 0x7fa57c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa593fff640 1 -- 172.21.15.146:0/1754514910 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57c003520 con 0x7fa57c0039b0 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa5988d5640 1 -- 172.21.15.146:0/1754514910 reap_dead start 2024-04-25T15:21:57.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa593fff640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa57c0039b0 0x7fa57c003e20 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c240 tx=0x7fa57c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d5bd8ff1ccf4ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584007cf0 con 0x7fa57c0039b0 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.756+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5840058d0 con 0x7fa57c0039b0 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa592ffd640 1 -- 172.21.15.146:0/1754514910 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584005d20 con 0x7fa57c0039b0 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 -- 172.21.15.146:0/1754514910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa57c0039b0 msgr2=0x7fa57c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa57c0039b0 0x7fa57c003e20 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c240 tx=0x7fa57c005830 comp rx=0 tx=0).stop 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 -- 172.21.15.146:0/1754514910 shutdown_connections 2024-04-25T15:21:57.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 --2- 172.21.15.146:0/1754514910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa57c0039b0 0x7fa57c003e20 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 -- 172.21.15.146:0/1754514910 >> 172.21.15.146:0/1754514910 conn(0x7fa5940b7490 msgr2=0x7fa5940b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:57.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 -- 172.21.15.146:0/1754514910 shutdown_connections 2024-04-25T15:21:57.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.757+0000 7fa59a338640 1 -- 172.21.15.146:0/1754514910 wait complete. 2024-04-25T15:21:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.759+0000 7fa59a338640 1 Processor -- start 2024-04-25T15:21:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.759+0000 7fa59a338640 1 -- start start 2024-04-25T15:21:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.760+0000 7fa59a338640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c0039b0 0x7fa5941f7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.760+0000 7fa59a338640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa594160d30 0x7fa5940b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.760+0000 7fa59a338640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 0x7fa59415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.760+0000 7fa59a338640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa59409b7b0 con 0x7fa5940b2690 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.760+0000 7fa59a338640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa59409b630 con 0x7fa57c0039b0 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.760+0000 7fa59a338640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa5940d2780 con 0x7fa594160d30 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 0x7fa59415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 0x7fa59415b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50834/0 (socket says 172.21.15.146:50834) 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 -- 172.21.15.146:0/2814248342 learned_addr learned my addr 172.21.15.146:0/2814248342 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5937fe640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa594160d30 0x7fa5940b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa593fff640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c0039b0 0x7fa5941f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 -- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa594160d30 msgr2=0x7fa5940b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa594160d30 0x7fa5940b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 -- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c0039b0 msgr2=0x7fa5941f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c0039b0 0x7fa5941f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 -- 172.21.15.146:0/2814248342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5940b2aa0 con 0x7fa5940b2690 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa593fff640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c0039b0 0x7fa5941f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa5988d5640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 0x7fa59415b7c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa594074550 tx=0x7fa58c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c003970 con 0x7fa5940b2690 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58c018070 con 0x7fa5940b2690 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5941f7560 con 0x7fa5940b2690 2024-04-25T15:21:57.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c003d70 con 0x7fa5940b2690 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.761+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5941f7cc0 con 0x7fa5940b2690 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.763+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa59406a0e0 con 0x7fa5940b2690 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.763+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa58c0050d0 con 0x7fa5940b2690 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.763+0000 7fa590ff9640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa580074540 0x7fa580076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.764+0000 7fa593fff640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa580074540 0x7fa580076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.764+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa58c0951b0 con 0x7fa5940b2690 2024-04-25T15:21:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.764+0000 7fa593fff640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa580074540 0x7fa580076a50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa584033000 tx=0x7fa584009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:57.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.768+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5940b2aa0 con 0x7fa5940b2690 2024-04-25T15:21:57.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.941+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa594071420 con 0x7fa5940b2690 2024-04-25T15:21:57.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.941+0000 7fa590ff9640 1 -- 172.21.15.146:0/2814248342 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fa58c05f7e0 con 0x7fa5940b2690 2024-04-25T15:21:57.943 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa580074540 msgr2=0x7fa580076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa580074540 0x7fa580076a50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa584033000 tx=0x7fa584009030 comp rx=0 tx=0).stop 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 msgr2=0x7fa59415b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 0x7fa59415b7c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa594074550 tx=0x7fa58c00ec10 comp rx=0 tx=0).stop 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 shutdown_connections 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa580074540 0x7fa580076a50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5940b2690 0x7fa59415b7c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa594160d30 0x7fa5940b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 --2- 172.21.15.146:0/2814248342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa57c0039b0 0x7fa5941f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:57.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 >> 172.21.15.146:0/2814248342 conn(0x7fa5940b7490 msgr2=0x7fa59406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:57.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 shutdown_connections 2024-04-25T15:21:57.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:57.943+0000 7fa59a338640 1 -- 172.21.15.146:0/2814248342 wait complete. 2024-04-25T15:21:57.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.067+0000 7f3f7fbee640 1 Processor -- start 2024-04-25T15:21:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.067+0000 7f3f7fbee640 1 -- start start 2024-04-25T15:21:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7fbee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f780b2cd0 0x7f3f780cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7fbee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3f780d2440 0x7f3f78071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7fbee640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f780cf7d0 con 0x7f3f780d1920 2024-04-25T15:21:58.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7fbee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3f7808da90 con 0x7f3f780b2cd0 2024-04-25T15:21:58.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7fbee640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3f780d0070 con 0x7f3f780d2440 2024-04-25T15:21:58.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7d98a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f780b2cd0 0x7f3f780cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.068+0000 7f3f7d98a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f780b2cd0 0x7f3f780cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50848/0 (socket says 172.21.15.146:50848) 2024-04-25T15:21:58.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7d98a640 1 -- 172.21.15.146:0/351459582 learned_addr learned my addr 172.21.15.146:0/351459582 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:58.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7e18b640 1 --2- 172.21.15.146:0/351459582 >> v2:172.21.15.177:3300/0 conn(0x7f3f780d2440 0x7f3f78071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7d98a640 1 -- 172.21.15.146:0/351459582 >> v2:172.21.15.177:3300/0 conn(0x7f3f780d2440 msgr2=0x7f3f78071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:58.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7d98a640 1 --2- 172.21.15.146:0/351459582 >> v2:172.21.15.177:3300/0 conn(0x7f3f780d2440 0x7f3f78071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7d98a640 1 -- 172.21.15.146:0/351459582 >> v1:172.21.15.177:6789/0 conn(0x7f3f780d1920 legacy=0x7f3f780d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7d98a640 1 -- 172.21.15.146:0/351459582 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f780cfab0 con 0x7f3f780b2cd0 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.069+0000 7f3f7e18b640 1 --2- 172.21.15.146:0/351459582 >> v2:172.21.15.177:3300/0 conn(0x7f3f780d2440 0x7f3f78071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7d98a640 1 --2- 172.21.15.146:0/351459582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780b2cd0 0x7f3f780cf0f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3f780b9870 tx=0x7f3f7402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d02da65d8bbe89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7c988640 1 -- 172.21.15.146:0/351459582 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74004030 con 0x7f3f780b2cd0 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7c988640 1 -- 172.21.15.146:0/351459582 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f740041d0 con 0x7f3f780b2cd0 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7c988640 1 -- 172.21.15.146:0/351459582 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74003610 con 0x7f3f780b2cd0 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7fbee640 1 -- 172.21.15.146:0/351459582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780b2cd0 msgr2=0x7f3f780cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/351459582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780b2cd0 0x7f3f780cf0f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3f780b9870 tx=0x7f3f7402a1c0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7fbee640 1 -- 172.21.15.146:0/351459582 shutdown_connections 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/351459582 >> v2:172.21.15.177:3300/0 conn(0x7f3f780d2440 0x7f3f78071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/351459582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780b2cd0 0x7f3f780cf0f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.070+0000 7f3f7fbee640 1 -- 172.21.15.146:0/351459582 >> 172.21.15.146:0/351459582 conn(0x7f3f780b52e0 msgr2=0x7f3f780b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.071+0000 7f3f7fbee640 1 -- 172.21.15.146:0/351459582 shutdown_connections 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.071+0000 7f3f7fbee640 1 -- 172.21.15.146:0/351459582 wait complete. 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.072+0000 7f3f7fbee640 1 Processor -- start 2024-04-25T15:21:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.072+0000 7f3f7fbee640 1 -- start start 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f780b2cd0 0x7f3f7815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 0x7f3f780cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f780d2440 0x7f3f780a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3f780721b0 con 0x7f3f780d1920 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3f780717e0 con 0x7f3f780b2cd0 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3f78072330 con 0x7f3f780d2440 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 0x7f3f780cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 0x7f3f780cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50864/0 (socket says 172.21.15.146:50864) 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 -- 172.21.15.146:0/1370527655 learned_addr learned my addr 172.21.15.146:0/1370527655 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7e18b640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f780d2440 0x7f3f780a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d98a640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f780b2cd0 0x7f3f7815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 -- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f780d2440 msgr2=0x7f3f780a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f780d2440 0x7f3f780a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 -- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f780b2cd0 msgr2=0x7f3f7815a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f780b2cd0 0x7f3f7815a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 -- 172.21.15.146:0/1370527655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f780cd670 con 0x7f3f780d1920 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7e18b640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f780d2440 0x7f3f780a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d98a640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f780b2cd0 0x7f3f7815a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7d189640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 0x7f3f780cd130 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800c3c0 tx=0x7f3f6800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680052d0 con 0x7f3f780d1920 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f68005490 con 0x7f3f780d1920 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f780caa40 con 0x7f3f780d1920 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68007260 con 0x7f3f780d1920 2024-04-25T15:21:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.074+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f780cb000 con 0x7f3f780d1920 2024-04-25T15:21:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.076+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7806a0e0 con 0x7f3f780d1920 2024-04-25T15:21:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.076+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f3f68007400 con 0x7f3f780d1920 2024-04-25T15:21:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.076+0000 7f3f6e7fc640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f5c074540 0x7f3f5c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.077+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f3f68094910 con 0x7f3f780d1920 2024-04-25T15:21:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.077+0000 7f3f7d98a640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f5c074540 0x7f3f5c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.077+0000 7f3f7d98a640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f5c074540 0x7f3f5c076a50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f74033000 tx=0x7f3f7402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:58.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.081+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f6805ef40 con 0x7f3f780d1920 2024-04-25T15:21:58.249 INFO:tasks.ceph.mds.d.smithi177.stderr:2024-04-25T15:21:58.247+0000 7ff855cb1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:21:58.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.256+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3f7806d3a0 con 0x7f3f780d1920 2024-04-25T15:21:58.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.257+0000 7f3f6e7fc640 1 -- 172.21.15.146:0/1370527655 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f3f6805e840 con 0x7f3f780d1920 2024-04-25T15:21:58.258 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:21:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f5c074540 msgr2=0x7f3f5c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f5c074540 0x7f3f5c076a50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f74033000 tx=0x7f3f7402e000 comp rx=0 tx=0).stop 2024-04-25T15:21:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 msgr2=0x7f3f780cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 0x7f3f780cd130 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800c3c0 tx=0x7f3f6800c7c0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 shutdown_connections 2024-04-25T15:21:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3f5c074540 0x7f3f5c076a50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f780d2440 0x7f3f780a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f780d1920 0x7f3f780cd130 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 --2- 172.21.15.146:0/1370527655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f780b2cd0 0x7f3f7815a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 >> 172.21.15.146:0/1370527655 conn(0x7f3f780b52e0 msgr2=0x7f3f780726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 shutdown_connections 2024-04-25T15:21:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.259+0000 7f3f7fbee640 1 -- 172.21.15.146:0/1370527655 wait complete. 2024-04-25T15:21:58.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:21:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.380+0000 7fa86064c640 1 Processor -- start 2024-04-25T15:21:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.380+0000 7fa86064c640 1 -- start start 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa86064c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa8580cc280 0x7fa858070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa86064c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa85806c8c0 0x7fa85806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa86064c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580cd490 con 0x7fa8581618f0 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa86064c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa85815bcd0 con 0x7fa8580cc280 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa86064c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa8580711d0 con 0x7fa85806c8c0 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa8580cc280 0x7fa858070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa8580cc280 0x7fa858070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50874/0 (socket says 172.21.15.146:50874) 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 -- 172.21.15.146:0/2095904601 learned_addr learned my addr 172.21.15.146:0/2095904601 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85ebe9640 1 --2- 172.21.15.146:0/2095904601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa85806c8c0 0x7fa85806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 -- 172.21.15.146:0/2095904601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa85806c8c0 msgr2=0x7fa85806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 --2- 172.21.15.146:0/2095904601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa85806c8c0 0x7fa85806ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 -- 172.21.15.146:0/2095904601 >> v1:172.21.15.177:6789/0 conn(0x7fa8581618f0 legacy=0x7fa858161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.382+0000 7fa85e3e8640 1 -- 172.21.15.146:0/2095904601 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8580cc880 con 0x7fa8580cc280 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.383+0000 7fa85e3e8640 1 --2- 172.21.15.146:0/2095904601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8580cc280 0x7fa858070c90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa854009e50 tx=0x7fa85402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a7611b331483ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.383+0000 7fa85d3e6640 1 -- 172.21.15.146:0/2095904601 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854036070 con 0x7fa8580cc280 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.383+0000 7fa85d3e6640 1 -- 172.21.15.146:0/2095904601 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa854031600 con 0x7fa8580cc280 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.383+0000 7fa85d3e6640 1 -- 172.21.15.146:0/2095904601 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8540319d0 con 0x7fa8580cc280 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 -- 172.21.15.146:0/2095904601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8580cc280 msgr2=0x7fa858070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 --2- 172.21.15.146:0/2095904601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8580cc280 0x7fa858070c90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa854009e50 tx=0x7fa85402a680 comp rx=0 tx=0).stop 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 -- 172.21.15.146:0/2095904601 shutdown_connections 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 --2- 172.21.15.146:0/2095904601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa85806c8c0 0x7fa85806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 --2- 172.21.15.146:0/2095904601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8580cc280 0x7fa858070c90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 -- 172.21.15.146:0/2095904601 >> 172.21.15.146:0/2095904601 conn(0x7fa8580ce0c0 msgr2=0x7fa8580ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:58.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 -- 172.21.15.146:0/2095904601 shutdown_connections 2024-04-25T15:21:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.384+0000 7fa86064c640 1 -- 172.21.15.146:0/2095904601 wait complete. 2024-04-25T15:21:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.386+0000 7fa86064c640 1 Processor -- start 2024-04-25T15:21:58.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.386+0000 7fa86064c640 1 -- start start 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa86064c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 0x7fa85814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa86064c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8580cc280 0x7fa858160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa86064c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8581618f0 0x7fa85815e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa86064c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa8580cd120 con 0x7fa85806c8c0 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa86064c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa8580cce40 con 0x7fa8581618f0 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa86064c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa8580b0940 con 0x7fa8580cc280 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa85e3e8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 0x7fa85814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa85e3e8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 0x7fa85814ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50878/0 (socket says 172.21.15.146:50878) 2024-04-25T15:21:58.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.387+0000 7fa85e3e8640 1 -- 172.21.15.146:0/3806638836 learned_addr learned my addr 172.21.15.146:0/3806638836 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85ebe9640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8581618f0 0x7fa85815e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85dbe7640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8580cc280 0x7fa858160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85e3e8640 1 -- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8580cc280 msgr2=0x7fa858160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85e3e8640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8580cc280 0x7fa858160030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85e3e8640 1 -- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8581618f0 msgr2=0x7fa85815e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85e3e8640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8581618f0 0x7fa85815e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85e3e8640 1 -- 172.21.15.146:0/3806638836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85815ed30 con 0x7fa85806c8c0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa85e3e8640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 0x7fa85814ff10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa854009f70 tx=0x7fa854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85403f020 con 0x7fa85806c8c0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa854037040 con 0x7fa85806c8c0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85815f550 con 0x7fa85806c8c0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa854036070 con 0x7fa85806c8c0 2024-04-25T15:21:58.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.388+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa858160570 con 0x7fa85806c8c0 2024-04-25T15:21:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.389+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8581f50d0 con 0x7fa85806c8c0 2024-04-25T15:21:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.390+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa854042020 con 0x7fa85806c8c0 2024-04-25T15:21:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.390+0000 7fa84effd640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa83c074540 0x7fa83c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:58.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.390+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa8540b5a40 con 0x7fa85806c8c0 2024-04-25T15:21:58.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.390+0000 7fa85dbe7640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa83c074540 0x7fa83c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:58.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.391+0000 7fa85dbe7640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa83c074540 0x7fa83c076a50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa8580ceb70 tx=0x7fa848009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:58.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.395+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa854049030 con 0x7fa85806c8c0 2024-04-25T15:21:58.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.584+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa85806a0e0 con 0x7fa85806c8c0 2024-04-25T15:21:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.584+0000 7fa84effd640 1 -- 172.21.15.146:0/3806638836 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7fa85407ff60 con 0x7fa85806c8c0 2024-04-25T15:21:58.585 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6835/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:21:58.585 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6833/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:21:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:21:58.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.586+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa83c074540 msgr2=0x7fa83c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.586+0000 7fa86064c640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa83c074540 0x7fa83c076a50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa8580ceb70 tx=0x7fa848009040 comp rx=0 tx=0).stop 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.586+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 msgr2=0x7fa85814ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.586+0000 7fa86064c640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 0x7fa85814ff10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa854009f70 tx=0x7fa854004a40 comp rx=0 tx=0).stop 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 shutdown_connections 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa83c074540 0x7fa83c076a50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8581618f0 0x7fa85815e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8580cc280 0x7fa858160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 --2- 172.21.15.146:0/3806638836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa85806c8c0 0x7fa85814ff10 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 >> 172.21.15.146:0/3806638836 conn(0x7fa8580ce0c0 msgr2=0x7fa85815c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 shutdown_connections 2024-04-25T15:21:58.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:58.587+0000 7fa86064c640 1 -- 172.21.15.146:0/3806638836 wait complete. 2024-04-25T15:21:58.599 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:21:58.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:58.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-25T15:21:58.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[42562]: fuse finished with error 0 and tester_r 0 2024-04-25T15:21:58.628 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:21:58.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:21:58.628 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:21:58.799 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:21:58.799 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:58.800 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:58.816 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:21:58.816 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:21:58.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:21:59.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.019+0000 7f02f660c640 1 Processor -- start 2024-04-25T15:21:59.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.019+0000 7f02f660c640 1 -- start start 2024-04-25T15:21:59.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f660c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f01645c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f660c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02f0164b00 0x7f02f00d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f660c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02f00b8eb0 con 0x7f02f006ee60 2024-04-25T15:21:59.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f660c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f02f0161840 con 0x7f02f0164b00 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f660c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f02f008d730 con 0x7f02f00b2ba0 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02f0164b00 0x7f02f00d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02f0164b00 0x7f02f00d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50882/0 (socket says 172.21.15.146:50882) 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 -- 172.21.15.146:0/1498564303 learned_addr learned my addr 172.21.15.146:0/1498564303 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 -- 172.21.15.146:0/1498564303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 msgr2=0x7f02f01645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02ef7fe640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f01645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f01645c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 -- 172.21.15.146:0/1498564303 >> v1:172.21.15.177:6789/0 conn(0x7f02f006ee60 legacy=0x7f02f0163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02f4ba9640 1 -- 172.21.15.146:0/1498564303 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f008ce00 con 0x7f02f0164b00 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.021+0000 7f02ef7fe640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f01645c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.022+0000 7f02f4ba9640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f0164b00 0x7f02f00d1b60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009e50 tx=0x7f02e802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244a73649693b96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.022+0000 7f02eeffd640 1 -- 172.21.15.146:0/1498564303 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8036070 con 0x7f02f0164b00 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.022+0000 7f02eeffd640 1 -- 172.21.15.146:0/1498564303 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e8031600 con 0x7f02f0164b00 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.022+0000 7f02eeffd640 1 -- 172.21.15.146:0/1498564303 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e80319d0 con 0x7f02f0164b00 2024-04-25T15:21:59.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 -- 172.21.15.146:0/1498564303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f0164b00 msgr2=0x7f02f00d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f0164b00 0x7f02f00d1b60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009e50 tx=0x7f02e802a6c0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 -- 172.21.15.146:0/1498564303 shutdown_connections 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f0164b00 0x7f02f00d1b60 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 --2- 172.21.15.146:0/1498564303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f01645c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 -- 172.21.15.146:0/1498564303 >> 172.21.15.146:0/1498564303 conn(0x7f02f00b52e0 msgr2=0x7f02f00b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 -- 172.21.15.146:0/1498564303 shutdown_connections 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.023+0000 7f02f660c640 1 -- 172.21.15.146:0/1498564303 wait complete. 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.024+0000 7f02f660c640 1 Processor -- start 2024-04-25T15:21:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.025+0000 7f02f660c640 1 -- start start 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f660c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 0x7f02f015dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f660c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f00b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f660c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f0164b00 0x7f02f00b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f660c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02f008d3c0 con 0x7f02f006ee60 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f660c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02f006f440 con 0x7f02f0164b00 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f660c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f02f008dbc0 con 0x7f02f00b2ba0 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 0x7f02f015dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 0x7f02f015dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50884/0 (socket says 172.21.15.146:50884) 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 -- 172.21.15.146:0/3145330117 learned_addr learned my addr 172.21.15.146:0/3145330117 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02ef7fe640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f00b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02f4ba9640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f0164b00 0x7f02f00b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 -- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 msgr2=0x7f02f00b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f00b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 -- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f0164b00 msgr2=0x7f02f00b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f0164b00 0x7f02f00b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 -- 172.21.15.146:0/3145330117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f01eb060 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.026+0000 7f02effff640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 0x7f02f015dc10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f02e000b3e0 tx=0x7f02e000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.027+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0015020 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.027+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e0011850 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.027+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f0162470 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.027+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0011c40 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.027+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f0162a30 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.028+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f006a0e0 con 0x7f02f006ee60 2024-04-25T15:21:59.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.028+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f02e0018020 con 0x7f02f006ee60 2024-04-25T15:21:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.029+0000 7f02ecff9640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f02dc074540 0x7f02dc076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.029+0000 7f02ef7fe640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f02dc074540 0x7f02dc076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.029+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f02e0095180 con 0x7f02f006ee60 2024-04-25T15:21:59.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.029+0000 7f02ef7fe640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f02dc074540 0x7f02dc076a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02f0073640 tx=0x7f02e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.033+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f02e005f7b0 con 0x7f02f006ee60 2024-04-25T15:21:59.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.208+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02f01648e0 con 0x7f02f006ee60 2024-04-25T15:21:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.210+0000 7f02ecff9640 1 -- 172.21.15.146:0/3145330117 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7f02e005f0b0 con 0x7f02f006ee60 2024-04-25T15:21:59.211 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:21:59.211 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f02dc074540 msgr2=0x7f02dc076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f02dc074540 0x7f02dc076a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02f0073640 tx=0x7f02e4008040 comp rx=0 tx=0).stop 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 msgr2=0x7f02f015dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 0x7f02f015dc10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f02e000b3e0 tx=0x7f02e000b8a0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 shutdown_connections 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f02dc074540 0x7f02dc076a50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f0164b00 0x7f02f00b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02f00b2ba0 0x7f02f00b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 --2- 172.21.15.146:0/3145330117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f006ee60 0x7f02f015dc10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 >> 172.21.15.146:0/3145330117 conn(0x7f02f00b52e0 msgr2=0x7f02f015e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 shutdown_connections 2024-04-25T15:21:59.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.212+0000 7f02f660c640 1 -- 172.21.15.146:0/3145330117 wait complete. 2024-04-25T15:21:59.224 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:21:59.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:21:59.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.335+0000 7fc5f9fee640 1 Processor -- start 2024-04-25T15:21:59.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.335+0000 7fc5f9fee640 1 -- start start 2024-04-25T15:21:59.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f9fee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc5f40b2cd0 0x7fc5f40d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f9fee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5f40d3300 0x7fc5f4071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f9fee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f4071f70 con 0x7fc5f4071b60 2024-04-25T15:21:59.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f9fee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f40d2260 con 0x7fc5f40d3300 2024-04-25T15:21:59.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f9fee640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f40d2540 con 0x7fc5f40b2cd0 2024-04-25T15:21:59.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f2ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5f40d3300 0x7fc5f4071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc5f4071b60 0x7fc5f408cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45194/0 (socket says 172.21.15.146:45194) 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f2ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5f40d3300 0x7fc5f4071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50894/0 (socket says 172.21.15.146:50894) 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f3fff640 1 -- 172.21.15.146:0/548886771 learned_addr learned my addr 172.21.15.146:0/548886771 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f37fe640 1 --2- 172.21.15.146:0/548886771 >> v2:172.21.15.177:3300/0 conn(0x7fc5f40b2cd0 0x7fc5f40d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f2ffd640 1 -- 172.21.15.146:0/548886771 >> v2:172.21.15.177:3300/0 conn(0x7fc5f40b2cd0 msgr2=0x7fc5f40d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f2ffd640 1 --2- 172.21.15.146:0/548886771 >> v2:172.21.15.177:3300/0 conn(0x7fc5f40b2cd0 0x7fc5f40d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f2ffd640 1 -- 172.21.15.146:0/548886771 >> v1:172.21.15.146:6789/0 conn(0x7fc5f4071b60 legacy=0x7fc5f408cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f2ffd640 1 -- 172.21.15.146:0/548886771 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f40d2820 con 0x7fc5f40d3300 2024-04-25T15:21:59.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.337+0000 7fc5f27fc640 1 -- 172.21.15.146:0/548886771 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 334611234 0 0) 0x7fc5ec001490 con 0x7fc5f4071b60 2024-04-25T15:21:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.338+0000 7fc5f2ffd640 1 --2- 172.21.15.146:0/548886771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f40d3300 0x7fc5f4071620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc5f408c9d0 tx=0x7fc5e802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165ea5f0e646a343 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.338+0000 7fc5f27fc640 1 -- 172.21.15.146:0/548886771 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8037070 con 0x7fc5f40d3300 2024-04-25T15:21:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.338+0000 7fc5f27fc640 1 -- 172.21.15.146:0/548886771 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e8032440 con 0x7fc5f40d3300 2024-04-25T15:21:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.338+0000 7fc5f27fc640 1 -- 172.21.15.146:0/548886771 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e802ae60 con 0x7fc5f40d3300 2024-04-25T15:21:59.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 -- 172.21.15.146:0/548886771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f40d3300 msgr2=0x7fc5f4071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/548886771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f40d3300 0x7fc5f4071620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc5f408c9d0 tx=0x7fc5e802a110 comp rx=0 tx=0).stop 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 -- 172.21.15.146:0/548886771 shutdown_connections 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/548886771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f40d3300 0x7fc5f4071620 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/548886771 >> v2:172.21.15.177:3300/0 conn(0x7fc5f40b2cd0 0x7fc5f40d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 -- 172.21.15.146:0/548886771 >> 172.21.15.146:0/548886771 conn(0x7fc5f40b52e0 msgr2=0x7fc5f40b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 -- 172.21.15.146:0/548886771 shutdown_connections 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.339+0000 7fc5f9fee640 1 -- 172.21.15.146:0/548886771 wait complete. 2024-04-25T15:21:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.341+0000 7fc5f9fee640 1 Processor -- start 2024-04-25T15:21:59.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.341+0000 7fc5f9fee640 1 -- start start 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f9fee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 0x7fc5f4160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f9fee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5f40b2cd0 0x7fc5f415f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f9fee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5f40d3300 0x7fc5f4162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f9fee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f40cfb70 con 0x7fc5f4071b60 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f9fee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f40d2de0 con 0x7fc5f40b2cd0 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f9fee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f408dbf0 con 0x7fc5f40d3300 2024-04-25T15:21:59.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 0x7fc5f4160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 0x7fc5f4160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50906/0 (socket says 172.21.15.146:50906) 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f37fe640 1 -- 172.21.15.146:0/4115468787 learned_addr learned my addr 172.21.15.146:0/4115468787 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f3fff640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5f40d3300 0x7fc5f4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.342+0000 7fc5f2ffd640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5f40b2cd0 0x7fc5f415f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f37fe640 1 -- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5f40d3300 msgr2=0x7fc5f4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f37fe640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5f40d3300 0x7fc5f4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f37fe640 1 -- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5f40b2cd0 msgr2=0x7fc5f415f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f37fe640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5f40b2cd0 0x7fc5f415f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f37fe640 1 -- 172.21.15.146:0/4115468787 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f4160bd0 con 0x7fc5f4071b60 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f3fff640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5f40d3300 0x7fc5f4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f2ffd640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5f40b2cd0 0x7fc5f415f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f37fe640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 0x7fc5f4160690 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400d800 tx=0x7fc5e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4005990 con 0x7fc5f4071b60 2024-04-25T15:21:59.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e4005b50 con 0x7fc5f4071b60 2024-04-25T15:21:59.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f41610e0 con 0x7fc5f4071b60 2024-04-25T15:21:59.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40096a0 con 0x7fc5f4071b60 2024-04-25T15:21:59.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.343+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f41616a0 con 0x7fc5f4071b60 2024-04-25T15:21:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.344+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f406a0e0 con 0x7fc5f4071b60 2024-04-25T15:21:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.345+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fc5e4009840 con 0x7fc5f4071b60 2024-04-25T15:21:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.345+0000 7fc5dbfff640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5e0074540 0x7fc5e0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.345+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc5e4095750 con 0x7fc5f4071b60 2024-04-25T15:21:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.345+0000 7fc5f2ffd640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5e0074540 0x7fc5e0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.346+0000 7fc5f2ffd640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5e0074540 0x7fc5e0076a50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8034000 tx=0x7fc5e802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.350+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5e405fcf0 con 0x7fc5f4071b60 2024-04-25T15:21:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.521+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc5f40688a0 con 0x7fc5f4071b60 2024-04-25T15:21:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.522+0000 7fc5dbfff640 1 -- 172.21.15.146:0/4115468787 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fc5e405f5f0 con 0x7fc5f4071b60 2024-04-25T15:21:59.523 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:21:59.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.523+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5e0074540 msgr2=0x7fc5e0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.523+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5e0074540 0x7fc5e0076a50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8034000 tx=0x7fc5e802e000 comp rx=0 tx=0).stop 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.523+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 msgr2=0x7fc5f4160690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.523+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 0x7fc5f4160690 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400d800 tx=0x7fc5e400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 shutdown_connections 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5e0074540 0x7fc5e0076a50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5f40d3300 0x7fc5f4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5f40b2cd0 0x7fc5f415f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 --2- 172.21.15.146:0/4115468787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5f4071b60 0x7fc5f4160690 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 >> 172.21.15.146:0/4115468787 conn(0x7fc5f40b52e0 msgr2=0x7fc5f40ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 shutdown_connections 2024-04-25T15:21:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.524+0000 7fc5f9fee640 1 -- 172.21.15.146:0/4115468787 wait complete. 2024-04-25T15:21:59.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:21:59.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.646+0000 7f25ea19e640 1 Processor -- start 2024-04-25T15:21:59.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.646+0000 7f25ea19e640 1 -- start start 2024-04-25T15:21:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25ea19e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f25e40b0fa0 0x7f25e415e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25ea19e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e40b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25ea19e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e40aee10 con 0x7f25e40d1e30 2024-04-25T15:21:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25ea19e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f25e40b13b0 con 0x7f25e40b0fa0 2024-04-25T15:21:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25ea19e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f25e40af0b0 con 0x7f25e40b3070 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25e2ffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e40b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25e37fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f25e40b0fa0 0x7f25e415e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25e3fff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f25e40d1e30 0x7f25e40d2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:39910/0 (socket says 172.21.15.146:39910) 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.647+0000 7f25e3fff640 1 -- 172.21.15.146:0/2507112101 learned_addr learned my addr 172.21.15.146:0/2507112101 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e27fc640 1 -- 172.21.15.146:0/2507112101 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3902746336 0 0) 0x7f25e40aee10 con 0x7f25e40d1e30 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e27fc640 1 -- 172.21.15.146:0/2507112101 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25cc003520 con 0x7f25e40d1e30 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e2ffd640 1 -- 172.21.15.146:0/2507112101 >> v2:172.21.15.177:3300/0 conn(0x7f25e40b0fa0 msgr2=0x7f25e415e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e2ffd640 1 --2- 172.21.15.146:0/2507112101 >> v2:172.21.15.177:3300/0 conn(0x7f25e40b0fa0 0x7f25e415e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e2ffd640 1 -- 172.21.15.146:0/2507112101 >> v1:172.21.15.177:6789/0 conn(0x7f25e40d1e30 legacy=0x7f25e40d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e2ffd640 1 -- 172.21.15.146:0/2507112101 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e40af390 con 0x7f25e40b3070 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.648+0000 7f25e37fe640 1 --2- 172.21.15.146:0/2507112101 >> v2:172.21.15.177:3300/0 conn(0x7f25e40b0fa0 0x7f25e415e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25e2ffd640 1 --2- 172.21.15.146:0/2507112101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e40b3480 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097a0 tx=0x7f25d802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88ebe2b949368c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25e27fc640 1 -- 172.21.15.146:0/2507112101 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d8037070 con 0x7f25e40b3070 2024-04-25T15:21:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25e27fc640 1 -- 172.21.15.146:0/2507112101 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d8032440 con 0x7f25e40b3070 2024-04-25T15:21:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25e27fc640 1 -- 172.21.15.146:0/2507112101 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d80327f0 con 0x7f25e40b3070 2024-04-25T15:21:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25ea19e640 1 -- 172.21.15.146:0/2507112101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 msgr2=0x7f25e40b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2507112101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e40b3480 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097a0 tx=0x7f25d802a180 comp rx=0 tx=0).stop 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25ea19e640 1 -- 172.21.15.146:0/2507112101 shutdown_connections 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2507112101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e40b3480 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2507112101 >> v2:172.21.15.177:3300/0 conn(0x7f25e40b0fa0 0x7f25e415e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.649+0000 7f25ea19e640 1 -- 172.21.15.146:0/2507112101 >> 172.21.15.146:0/2507112101 conn(0x7f25e409b2b0 msgr2=0x7f25e40b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.650+0000 7f25ea19e640 1 -- 172.21.15.146:0/2507112101 shutdown_connections 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.650+0000 7f25ea19e640 1 -- 172.21.15.146:0/2507112101 wait complete. 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.651+0000 7f25ea19e640 1 Processor -- start 2024-04-25T15:21:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.651+0000 7f25ea19e640 1 -- start start 2024-04-25T15:21:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25ea19e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25e40b0fa0 0x7f25e415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25ea19e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25ea19e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 0x7f25e406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25ea19e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f25e40cf650 con 0x7f25e40d1e30 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25ea19e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f25e40af950 con 0x7f25e40b0fa0 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25ea19e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f25e40cf7d0 con 0x7f25e40b3070 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 0x7f25e406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 0x7f25e406ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50918/0 (socket says 172.21.15.146:50918) 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 -- 172.21.15.146:0/2301994607 learned_addr learned my addr 172.21.15.146:0/2301994607 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e2ffd640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e37fe640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25e40b0fa0 0x7f25e415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 -- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 msgr2=0x7f25e406c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e406c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 -- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25e40b0fa0 msgr2=0x7f25e415c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25e40b0fa0 0x7f25e415c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 -- 172.21.15.146:0/2301994607 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e406d620 con 0x7f25e40d1e30 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e2ffd640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e406c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e37fe640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25e40b0fa0 0x7f25e415c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.653+0000 7f25e3fff640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 0x7f25e406ce10 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00caa0 tx=0x7f25dc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.654+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc017070 con 0x7f25e40d1e30 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.654+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25dc00ac00 con 0x7f25e40d1e30 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.654+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc012400 con 0x7f25e40d1e30 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.654+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e409be80 con 0x7f25e40d1e30 2024-04-25T15:21:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.654+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e409c440 con 0x7f25e40d1e30 2024-04-25T15:21:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.655+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e406a0e0 con 0x7f25e40d1e30 2024-04-25T15:21:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.655+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f25dc01a070 con 0x7f25e40d1e30 2024-04-25T15:21:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.656+0000 7f25cbfff640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25d0074540 0x7f25d0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.656+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f25dc0950c0 con 0x7f25e40d1e30 2024-04-25T15:21:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.656+0000 7f25e37fe640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25d0074540 0x7f25d0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.656+0000 7f25e37fe640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25d0074540 0x7f25d0076a50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f25e4161bb0 tx=0x7f25d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.661+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f25dc05f010 con 0x7f25e40d1e30 2024-04-25T15:21:59.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.835+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f25e40cc340 con 0x7f25e40d1e30 2024-04-25T15:21:59.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.836+0000 7f25cbfff640 1 -- 172.21.15.146:0/2301994607 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f25dc063310 con 0x7f25e40d1e30 2024-04-25T15:21:59.836 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25d0074540 msgr2=0x7f25d0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25d0074540 0x7f25d0076a50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f25e4161bb0 tx=0x7f25d4008040 comp rx=0 tx=0).stop 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 msgr2=0x7f25e406ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 0x7f25e406ce10 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00caa0 tx=0x7f25dc00cf60 comp rx=0 tx=0).stop 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 shutdown_connections 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f25d0074540 0x7f25d0076a50 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e40d1e30 0x7f25e406ce10 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25e40b3070 0x7f25e406c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 --2- 172.21.15.146:0/2301994607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25e40b0fa0 0x7f25e415c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 >> 172.21.15.146:0/2301994607 conn(0x7f25e409b2b0 msgr2=0x7f25e4161300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 shutdown_connections 2024-04-25T15:21:59.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.838+0000 7f25ea19e640 1 -- 172.21.15.146:0/2301994607 wait complete. 2024-04-25T15:21:59.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:21:59.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.959+0000 7f0aa2586640 1 Processor -- start 2024-04-25T15:21:59.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.960+0000 7f0aa2586640 1 -- start start 2024-04-25T15:21:59.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0aa2586640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a9c070b20 0x7f0a9c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0aa2586640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0a9c06daa0 0x7f0a9c161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0aa2586640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9c06deb0 con 0x7f0a9c162180 2024-04-25T15:21:59.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0aa2586640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9c15ed80 con 0x7f0a9c070b20 2024-04-25T15:21:59.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0aa2586640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9c15e1f0 con 0x7f0a9c06daa0 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a9c070b20 0x7f0a9c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a9c070b20 0x7f0a9c06d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50930/0 (socket says 172.21.15.146:50930) 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 -- 172.21.15.146:0/30219697 learned_addr learned my addr 172.21.15.146:0/30219697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9b7fe640 1 --2- 172.21.15.146:0/30219697 >> v2:172.21.15.177:3300/0 conn(0x7f0a9c06daa0 0x7f0a9c161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 -- 172.21.15.146:0/30219697 >> v2:172.21.15.177:3300/0 conn(0x7f0a9c06daa0 msgr2=0x7f0a9c161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 --2- 172.21.15.146:0/30219697 >> v2:172.21.15.177:3300/0 conn(0x7f0a9c06daa0 0x7f0a9c161c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 -- 172.21.15.146:0/30219697 >> v1:172.21.15.177:6789/0 conn(0x7f0a9c162180 legacy=0x7f0a9c157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:21:59.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.961+0000 7f0a9bfff640 1 -- 172.21.15.146:0/30219697 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c15e450 con 0x7f0a9c070b20 2024-04-25T15:21:59.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.962+0000 7f0a9bfff640 1 --2- 172.21.15.146:0/30219697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c070b20 0x7f0a9c06d560 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009870 tx=0x7f0a8c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cad90f41baa6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.962+0000 7f0a9affd640 1 -- 172.21.15.146:0/30219697 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c037070 con 0x7f0a9c070b20 2024-04-25T15:21:59.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.962+0000 7f0a9affd640 1 -- 172.21.15.146:0/30219697 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c032440 con 0x7f0a9c070b20 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.962+0000 7f0a9affd640 1 -- 172.21.15.146:0/30219697 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c032810 con 0x7f0a9c070b20 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 -- 172.21.15.146:0/30219697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c070b20 msgr2=0x7f0a9c06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 --2- 172.21.15.146:0/30219697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c070b20 0x7f0a9c06d560 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009870 tx=0x7f0a8c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 -- 172.21.15.146:0/30219697 shutdown_connections 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 --2- 172.21.15.146:0/30219697 >> v2:172.21.15.177:3300/0 conn(0x7f0a9c06daa0 0x7f0a9c161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 --2- 172.21.15.146:0/30219697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c070b20 0x7f0a9c06d560 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 -- 172.21.15.146:0/30219697 >> 172.21.15.146:0/30219697 conn(0x7f0a9c0d1c10 msgr2=0x7f0a9c0733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 -- 172.21.15.146:0/30219697 shutdown_connections 2024-04-25T15:21:59.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.963+0000 7f0aa2586640 1 -- 172.21.15.146:0/30219697 wait complete. 2024-04-25T15:21:59.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.965+0000 7f0aa2586640 1 Processor -- start 2024-04-25T15:21:59.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.965+0000 7f0aa2586640 1 -- start start 2024-04-25T15:21:59.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa2586640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a9c06daa0 0x7f0a9c1e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa2586640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a9c070b20 0x7f0a9c14c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa2586640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 0x7f0a9c1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa2586640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c162590 con 0x7f0a9c162180 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa2586640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c15ea10 con 0x7f0a9c070b20 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa2586640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c070f30 con 0x7f0a9c06daa0 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa0b23640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 0x7f0a9c1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa0b23640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 0x7f0a9c1554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50942/0 (socket says 172.21.15.146:50942) 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0aa0b23640 1 -- 172.21.15.146:0/236856010 learned_addr learned my addr 172.21.15.146:0/236856010 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0a9b7fe640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a9c070b20 0x7f0a9c14c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.966+0000 7f0a9bfff640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a9c06daa0 0x7f0a9c1e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa0b23640 1 -- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a9c06daa0 msgr2=0x7f0a9c1e8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa0b23640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a9c06daa0 0x7f0a9c1e8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa0b23640 1 -- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a9c070b20 msgr2=0x7f0a9c14c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa0b23640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a9c070b20 0x7f0a9c14c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa0b23640 1 -- 172.21.15.146:0/236856010 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c1e8080 con 0x7f0a9c162180 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa0b23640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 0x7f0a9c1554b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400b970 tx=0x7f0a9400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94019070 con 0x7f0a9c162180 2024-04-25T15:21:59.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9400cb40 con 0x7f0a9c162180 2024-04-25T15:21:59.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c1e8360 con 0x7f0a9c162180 2024-04-25T15:21:59.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94014440 con 0x7f0a9c162180 2024-04-25T15:21:59.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.967+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c1e8920 con 0x7f0a9c162180 2024-04-25T15:21:59.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.968+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a9c06a0e0 con 0x7f0a9c162180 2024-04-25T15:21:59.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.969+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f0a9401c070 con 0x7f0a9c162180 2024-04-25T15:21:59.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.969+0000 7f0a98ff9640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0a88074540 0x7f0a88076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:21:59.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.969+0000 7f0a9bfff640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0a88074540 0x7f0a88076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:21:59.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.969+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0a94095070 con 0x7f0a9c162180 2024-04-25T15:21:59.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.970+0000 7f0a9bfff640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0a88074540 0x7f0a88076a50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c034000 tx=0x7f0a8c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:21:59.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:21:59.974+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a9405efd0 con 0x7f0a9c162180 2024-04-25T15:22:00.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.151+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a9c074430 con 0x7f0a9c162180 2024-04-25T15:22:00.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.152+0000 7f0a98ff9640 1 -- 172.21.15.146:0/236856010 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f0a940632d0 con 0x7f0a9c162180 2024-04-25T15:22:00.153 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0a88074540 msgr2=0x7f0a88076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0a88074540 0x7f0a88076a50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c034000 tx=0x7f0a8c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 msgr2=0x7f0a9c1554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 0x7f0a9c1554b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400b970 tx=0x7f0a9400be30 comp rx=0 tx=0).stop 2024-04-25T15:22:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 shutdown_connections 2024-04-25T15:22:00.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0a88074540 0x7f0a88076a50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a9c162180 0x7f0a9c1554b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a9c070b20 0x7f0a9c14c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 --2- 172.21.15.146:0/236856010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a9c06daa0 0x7f0a9c1e8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 >> 172.21.15.146:0/236856010 conn(0x7f0a9c0d1c10 msgr2=0x7f0a9c15ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 shutdown_connections 2024-04-25T15:22:00.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.154+0000 7f0aa2586640 1 -- 172.21.15.146:0/236856010 wait complete. 2024-04-25T15:22:00.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:00.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.274+0000 7f4bf48a8640 1 Processor -- start 2024-04-25T15:22:00.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.274+0000 7f4bf48a8640 1 -- start start 2024-04-25T15:22:00.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.275+0000 7f4bf48a8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.275+0000 7f4bf48a8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf00b5b90 con 0x7f4bf0162590 2024-04-25T15:22:00.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.275+0000 7f4bf48a8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf00d1be0 con 0x7f4bf0161980 2024-04-25T15:22:00.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.275+0000 7f4bf48a8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf00d27f0 con 0x7f4bf00b5240 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.275+0000 7f4be7fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4bf0162590 0x7f4bf00d2c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45208/0 (socket says 172.21.15.146:45208) 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.275+0000 7f4be7fff640 1 -- 172.21.15.146:0/2297729448 learned_addr learned my addr 172.21.15.146:0/2297729448 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4beed9d640 1 --2- 172.21.15.146:0/2297729448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765300963 0 0) 0x7f4bd8001110 con 0x7f4bf0162590 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bdc003520 con 0x7f4bf0162590 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450879571 0 0) 0x7f4bf00d1be0 con 0x7f4bf0161980 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bd8001110 con 0x7f4bf0161980 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4beed9d640 1 -- 172.21.15.146:0/2297729448 >> v1:172.21.15.177:6789/0 conn(0x7f4bf0161980 legacy=0x7f4bf00d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4beed9d640 1 -- 172.21.15.146:0/2297729448 >> v1:172.21.15.146:6789/0 conn(0x7f4bf0162590 legacy=0x7f4bf00d2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.276+0000 7f4beed9d640 1 -- 172.21.15.146:0/2297729448 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf00d1ec0 con 0x7f4bf00b5240 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.277+0000 7f4beed9d640 1 --2- 172.21.15.146:0/2297729448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00b5650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009dc0 tx=0x7f4be802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fc0c94314b87cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.277+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8007740 con 0x7f4bf00b5240 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.277+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be80078e0 con 0x7f4bf00b5240 2024-04-25T15:22:00.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.277+0000 7f4bedd9b640 1 -- 172.21.15.146:0/2297729448 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8007cb0 con 0x7f4bf00b5240 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.277+0000 7f4bf48a8640 1 -- 172.21.15.146:0/2297729448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 msgr2=0x7f4bf00b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.277+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/2297729448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00b5650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4be8009dc0 tx=0x7f4be802a6d0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.278+0000 7f4bf48a8640 1 -- 172.21.15.146:0/2297729448 shutdown_connections 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.278+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/2297729448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00b5650 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.278+0000 7f4bf48a8640 1 -- 172.21.15.146:0/2297729448 >> 172.21.15.146:0/2297729448 conn(0x7f4bf00d0240 msgr2=0x7f4bf00d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.278+0000 7f4bf48a8640 1 -- 172.21.15.146:0/2297729448 shutdown_connections 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.278+0000 7f4bf48a8640 1 -- 172.21.15.146:0/2297729448 wait complete. 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.279+0000 7f4bf48a8640 1 Processor -- start 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.280+0000 7f4bf48a8640 1 -- start start 2024-04-25T15:22:00.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bf48a8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bf48a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bf0161980 0x7f4bf015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bf48a8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 0x7f4bf015c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bf48a8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf00d3390 con 0x7f4bf0162590 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bf48a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf00d2480 con 0x7f4bf0161980 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bf48a8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf00b6c20 con 0x7f4bf00b5240 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 0x7f4bf015c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 0x7f4bf015c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50958/0 (socket says 172.21.15.146:50958) 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 -- 172.21.15.146:0/1301890239 learned_addr learned my addr 172.21.15.146:0/1301890239 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4be7fff640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bf0161980 0x7f4bf015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bee59c640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 -- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 msgr2=0x7f4bf00c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 -- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bf0161980 msgr2=0x7f4bf015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bf0161980 0x7f4bf015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 -- 172.21.15.146:0/1301890239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf015a1e0 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4bee59c640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4be7fff640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bf0161980 0x7f4bf015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4beed9d640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 0x7f4bf015c8f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4be802acc0 tx=0x7f4be8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.281+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be803f070 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.282+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be8031bc0 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.282+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf015a4c0 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.282+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8009450 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.282+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf015aa80 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.283+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf00c69c0 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.283+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f4be803b410 con 0x7f4bf0162590 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.284+0000 7f4be6ffd640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4bd0074540 0x7f4bd0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.284+0000 7f4bee59c640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4bd0074540 0x7f4bd0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.284+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f4be80b6bc0 con 0x7f4bf0162590 2024-04-25T15:22:00.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.284+0000 7f4bee59c640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4bd0074540 0x7f4bd0076a50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4be0001610 tx=0x7f4be0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:00.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.289+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4bf00c69c0 con 0x7f4bf0162590 2024-04-25T15:22:00.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.474+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4bf01599b0 con 0x7f4bf0162590 2024-04-25T15:22:00.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.474+0000 7f4be6ffd640 1 -- 172.21.15.146:0/1301890239 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f4bf01599b0 con 0x7f4bf0162590 2024-04-25T15:22:00.475 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6835/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:22:00.475 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6833/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:22:00.476 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:22:00.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4bd0074540 msgr2=0x7f4bd0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4bd0074540 0x7f4bd0076a50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4be0001610 tx=0x7f4be0009040 comp rx=0 tx=0).stop 2024-04-25T15:22:00.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 msgr2=0x7f4bf015c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 0x7f4bf015c8f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4be802acc0 tx=0x7f4be8004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 shutdown_connections 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4bd0074540 0x7f4bd0076a50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bf0162590 0x7f4bf015c8f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bf0161980 0x7f4bf015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 --2- 172.21.15.146:0/1301890239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4bf00b5240 0x7f4bf00c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 >> 172.21.15.146:0/1301890239 conn(0x7f4bf00d0240 msgr2=0x7f4bf00cea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 shutdown_connections 2024-04-25T15:22:00.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.476+0000 7f4bf48a8640 1 -- 172.21.15.146:0/1301890239 wait complete. 2024-04-25T15:22:00.491 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:22:00.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:00.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-25T15:22:00.514 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[53771]: fuse finished with error 0 and tester_r 0 2024-04-25T15:22:00.515 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:22:00.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:00.516 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T15:22:00.683 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:22:00.683 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:00.683 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:00.702 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:00.702 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:00.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:00.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.867+0000 7f53fefd6640 1 Processor -- start 2024-04-25T15:22:00.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.867+0000 7f53fefd6640 1 -- start start 2024-04-25T15:22:00.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53fefd6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53f80cfc60 0x7f53f80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53fefd6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f80b4f20 con 0x7f53f80d2c50 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53fefd6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f80b3a70 con 0x7f53f80b4b10 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53fefd6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f53f80b3d50 con 0x7f53f80cfc60 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f7fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53f80cfc60 0x7f53f80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f7fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53f80cfc60 0x7f53f80d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43504/0 (socket says 172.21.15.146:43504) 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53fd573640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f53f80d2c50 0x7f53f80af490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52124/0 (socket says 172.21.15.146:52124) 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f7fff640 1 -- 172.21.15.146:0/2609463946 learned_addr learned my addr 172.21.15.146:0/2609463946 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f7fff640 1 -- 172.21.15.146:0/2609463946 >> v1:172.21.15.177:6789/0 conn(0x7f53f80b4b10 legacy=0x7f53f80cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f7fff640 1 -- 172.21.15.146:0/2609463946 >> v1:172.21.15.146:6789/0 conn(0x7f53f80d2c50 legacy=0x7f53f80af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f77fe640 1 -- 172.21.15.146:0/2609463946 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007134548 0 0) 0x7f53f80b4f20 con 0x7f53f80d2c50 2024-04-25T15:22:00.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.869+0000 7f53f7fff640 1 -- 172.21.15.146:0/2609463946 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f80b4030 con 0x7f53f80cfc60 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.870+0000 7f53f7fff640 1 --2- 172.21.15.146:0/2609463946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f80cfc60 0x7f53f80d2710 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009870 tx=0x7f53ec02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80afb8749e313eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.870+0000 7f53f77fe640 1 -- 172.21.15.146:0/2609463946 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec037070 con 0x7f53f80cfc60 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.870+0000 7f53f77fe640 1 -- 172.21.15.146:0/2609463946 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53ec032440 con 0x7f53f80cfc60 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.870+0000 7f53f77fe640 1 -- 172.21.15.146:0/2609463946 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec032810 con 0x7f53f80cfc60 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 -- 172.21.15.146:0/2609463946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f80cfc60 msgr2=0x7f53f80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 --2- 172.21.15.146:0/2609463946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f80cfc60 0x7f53f80d2710 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009870 tx=0x7f53ec02a210 comp rx=0 tx=0).stop 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 -- 172.21.15.146:0/2609463946 shutdown_connections 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 --2- 172.21.15.146:0/2609463946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f80cfc60 0x7f53f80d2710 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 -- 172.21.15.146:0/2609463946 >> 172.21.15.146:0/2609463946 conn(0x7f53f80b73b0 msgr2=0x7f53f80b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:00.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 -- 172.21.15.146:0/2609463946 shutdown_connections 2024-04-25T15:22:00.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.871+0000 7f53fefd6640 1 -- 172.21.15.146:0/2609463946 wait complete. 2024-04-25T15:22:00.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.872+0000 7f53fefd6640 1 Processor -- start 2024-04-25T15:22:00.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.872+0000 7f53fefd6640 1 -- start start 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fefd6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53f80b4b10 0x7f53f80c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fefd6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53f80d2c50 0x7f53f81f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fefd6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 0x7f53f8150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fefd6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f53f80d3060 con 0x7f53f815bdb0 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fefd6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53f80b45f0 con 0x7f53f80d2c50 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fefd6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f53f80cefc0 con 0x7f53f80b4b10 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 0x7f53f8150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 0x7f53f8150980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43508/0 (socket says 172.21.15.146:43508) 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 -- 172.21.15.146:0/3815301735 learned_addr learned my addr 172.21.15.146:0/3815301735 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53f7fff640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53f80d2c50 0x7f53f81f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fcd72640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53f80b4b10 0x7f53f80c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 -- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53f80b4b10 msgr2=0x7f53f80c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53f80b4b10 0x7f53f80c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 -- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53f80d2c50 msgr2=0x7f53f81f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53f80d2c50 0x7f53f81f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 -- 172.21.15.146:0/3815301735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f815fc60 con 0x7f53f815bdb0 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.874+0000 7f53fd573640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 0x7f53f8150980 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c780 tx=0x7f53f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.875+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f0018070 con 0x7f53f815bdb0 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.875+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f000ce30 con 0x7f53f815bdb0 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.875+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f000a840 con 0x7f53f815bdb0 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.875+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f815ff40 con 0x7f53f815bdb0 2024-04-25T15:22:00.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.875+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f8160500 con 0x7f53f815bdb0 2024-04-25T15:22:00.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.876+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f806a0e0 con 0x7f53f815bdb0 2024-04-25T15:22:00.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.876+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f53f001b070 con 0x7f53f815bdb0 2024-04-25T15:22:00.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.877+0000 7f53f57fa640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f53e4074540 0x7f53e4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:00.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.877+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f53f0094930 con 0x7f53f815bdb0 2024-04-25T15:22:00.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.877+0000 7f53fcd72640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f53e4074540 0x7f53e4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:00.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.877+0000 7f53fcd72640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f53e4074540 0x7f53e4076a50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f53f80cd1b0 tx=0x7f53e8009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:00.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:00.881+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53f005e880 con 0x7f53f815bdb0 2024-04-25T15:22:01.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.060+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53f80c6740 con 0x7f53f815bdb0 2024-04-25T15:22:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.061+0000 7f53f57fa640 1 -- 172.21.15.146:0/3815301735 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7f53f0062b80 con 0x7f53f815bdb0 2024-04-25T15:22:01.062 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:01.062 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f53e4074540 msgr2=0x7f53e4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f53e4074540 0x7f53e4076a50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f53f80cd1b0 tx=0x7f53e8009060 comp rx=0 tx=0).stop 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 msgr2=0x7f53f8150980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 0x7f53f8150980 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f53f000c780 tx=0x7f53f000cc40 comp rx=0 tx=0).stop 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 shutdown_connections 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f53e4074540 0x7f53e4076a50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53f815bdb0 0x7f53f8150980 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53f80d2c50 0x7f53f81f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 --2- 172.21.15.146:0/3815301735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53f80b4b10 0x7f53f80c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 >> 172.21.15.146:0/3815301735 conn(0x7f53f80b73b0 msgr2=0x7f53f80711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:01.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 shutdown_connections 2024-04-25T15:22:01.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.063+0000 7f53fefd6640 1 -- 172.21.15.146:0/3815301735 wait complete. 2024-04-25T15:22:01.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:01.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.186+0000 7f2dd8442640 1 Processor -- start 2024-04-25T15:22:01.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.186+0000 7f2dd8442640 1 -- start start 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd8442640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd00d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd8442640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2dd00d2cc0 0x7f2dd00ced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd8442640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd00d30d0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd8442640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f2dd00b2b60 con 0x7f2dd00d2cc0 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd8442640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd00b2d90 con 0x7f2dd00b3940 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd69df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2dd00d2cc0 0x7f2dd00ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd59dd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2dd00b3d50 0x7f2dd00ce650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52138/0 (socket says 172.21.15.146:52138) 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd59dd640 1 -- 172.21.15.146:0/4145517057 learned_addr learned my addr 172.21.15.146:0/4145517057 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd69df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2dd00d2cc0 0x7f2dd00ced60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43516/0 (socket says 172.21.15.146:43516) 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd61de640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd00d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd69df640 1 -- 172.21.15.146:0/4145517057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 msgr2=0x7f2dd00d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd69df640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd00d2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd69df640 1 -- 172.21.15.146:0/4145517057 >> v1:172.21.15.146:6789/0 conn(0x7f2dd00b3d50 legacy=0x7f2dd00ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.187+0000 7f2dd51dc640 1 -- 172.21.15.146:0/4145517057 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192615512 0 0) 0x7f2dc0001110 con 0x7f2dd00b3d50 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.188+0000 7f2dd69df640 1 -- 172.21.15.146:0/4145517057 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd00b3050 con 0x7f2dd00d2cc0 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.188+0000 7f2dd61de640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd00d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.188+0000 7f2dd69df640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00ced60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80097a0 tx=0x7f2dc802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c0adbd249a9694 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.188+0000 7f2dd51dc640 1 -- 172.21.15.146:0/4145517057 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8037070 con 0x7f2dd00d2cc0 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.188+0000 7f2dd51dc640 1 -- 172.21.15.146:0/4145517057 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc802ae50 con 0x7f2dd00d2cc0 2024-04-25T15:22:01.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.188+0000 7f2dd51dc640 1 -- 172.21.15.146:0/4145517057 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8032660 con 0x7f2dd00d2cc0 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 -- 172.21.15.146:0/4145517057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00d2cc0 msgr2=0x7f2dd00ced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00ced60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80097a0 tx=0x7f2dc802a100 comp rx=0 tx=0).stop 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 -- 172.21.15.146:0/4145517057 shutdown_connections 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00ced60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 --2- 172.21.15.146:0/4145517057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd00d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 -- 172.21.15.146:0/4145517057 >> 172.21.15.146:0/4145517057 conn(0x7f2dd00b7490 msgr2=0x7f2dd00b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:01.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 -- 172.21.15.146:0/4145517057 shutdown_connections 2024-04-25T15:22:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.189+0000 7f2dd8442640 1 -- 172.21.15.146:0/4145517057 wait complete. 2024-04-25T15:22:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.191+0000 7f2dd8442640 1 Processor -- start 2024-04-25T15:22:01.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.191+0000 7f2dd8442640 1 -- start start 2024-04-25T15:22:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd8442640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd0158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd8442640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 0x7f2dd00ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd8442640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd8442640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd006f8a0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd8442640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd006f720 con 0x7f2dd00d2cc0 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd8442640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd009b630 con 0x7f2dd00b3940 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd59dd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 0x7f2dd00ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd59dd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 0x7f2dd00ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43524/0 (socket says 172.21.15.146:43524) 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd59dd640 1 -- 172.21.15.146:0/2593800291 learned_addr learned my addr 172.21.15.146:0/2593800291 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd61de640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd0158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.192+0000 7f2dd69df640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd59dd640 1 -- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 msgr2=0x7f2dd0158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd59dd640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd0158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd59dd640 1 -- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dd00d2cc0 msgr2=0x7f2dd00caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd59dd640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd59dd640 1 -- 172.21.15.146:0/2593800291 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd00c74d0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd59dd640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 0x7f2dd00ca660 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c410 tx=0x7f2dc000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0017070 con 0x7f2dd00b3d50 2024-04-25T15:22:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc0005ce0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00072f0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd00c77b0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.193+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd00c7d70 con 0x7f2dd00b3d50 2024-04-25T15:22:01.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.194+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd00cb180 con 0x7f2dd00b3d50 2024-04-25T15:22:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.195+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f2dc001a070 con 0x7f2dd00b3d50 2024-04-25T15:22:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.195+0000 7f2dc6ffd640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2db4074570 0x7f2db4076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.195+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f2dc0094740 con 0x7f2dd00b3d50 2024-04-25T15:22:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.195+0000 7f2dd61de640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2db4074570 0x7f2db4076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.196+0000 7f2dd61de640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2db4074570 0x7f2db4076a80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00f500 tx=0x7f2dcc007770 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.200+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2dd00d30d0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.373+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2dd006a0e0 con 0x7f2dd00b3d50 2024-04-25T15:22:01.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.373+0000 7f2dc6ffd640 1 -- 172.21.15.146:0/2593800291 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f2dc005e690 con 0x7f2dd00b3d50 2024-04-25T15:22:01.374 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:01.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2db4074570 msgr2=0x7f2db4076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2db4074570 0x7f2db4076a80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00f500 tx=0x7f2dcc007770 comp rx=0 tx=0).stop 2024-04-25T15:22:01.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 msgr2=0x7f2dd00ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 0x7f2dd00ca660 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c410 tx=0x7f2dc000c8d0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 shutdown_connections 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2db4074570 0x7f2db4076a80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dd00d2cc0 0x7f2dd00caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00b3d50 0x7f2dd00ca660 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 --2- 172.21.15.146:0/2593800291 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2dd00b3940 0x7f2dd0158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 >> 172.21.15.146:0/2593800291 conn(0x7f2dd00b7490 msgr2=0x7f2dd0070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 shutdown_connections 2024-04-25T15:22:01.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.375+0000 7f2dd8442640 1 -- 172.21.15.146:0/2593800291 wait complete. 2024-04-25T15:22:01.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:01.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.497+0000 7f4090a40640 1 Processor -- start 2024-04-25T15:22:01.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.497+0000 7f4090a40640 1 -- start start 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4090a40640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f408c0ccd40 0x7f408c09bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4090a40640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0d2680 0x7f408c0d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4090a40640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f408c0b2400 0x7f408c0b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4090a40640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f408c0d2fd0 con 0x7f408c0b2400 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4090a40640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f408c0b1960 con 0x7f408c0ccd40 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4090a40640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f408c0b3ce0 con 0x7f408c0d2680 2024-04-25T15:22:01.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f408c0b2400 0x7f408c0b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f408c0b2400 0x7f408c0b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43530/0 (socket says 172.21.15.146:43530) 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 -- 172.21.15.146:0/153247849 learned_addr learned my addr 172.21.15.146:0/153247849 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408a59c640 1 --2- 172.21.15.146:0/153247849 >> v2:172.21.15.177:3300/0 conn(0x7f408c0ccd40 0x7f408c09bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f4089d9b640 1 --2- 172.21.15.146:0/153247849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0d2680 0x7f408c0d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 -- 172.21.15.146:0/153247849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0d2680 msgr2=0x7f408c0d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 --2- 172.21.15.146:0/153247849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0d2680 0x7f408c0d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 -- 172.21.15.146:0/153247849 >> v2:172.21.15.177:3300/0 conn(0x7f408c0ccd40 msgr2=0x7f408c09bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 --2- 172.21.15.146:0/153247849 >> v2:172.21.15.177:3300/0 conn(0x7f408c0ccd40 0x7f408c09bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.499+0000 7f408ad9d640 1 -- 172.21.15.146:0/153247849 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c0b1c00 con 0x7f408c0b2400 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.500+0000 7f408ad9d640 1 --2- 172.21.15.146:0/153247849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0b2400 0x7f408c0b2810 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f408c0d2650 tx=0x7f408402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d919ddf6fe150ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.500+0000 7f408959a640 1 -- 172.21.15.146:0/153247849 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084036070 con 0x7f408c0b2400 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.500+0000 7f408959a640 1 -- 172.21.15.146:0/153247849 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4084031600 con 0x7f408c0b2400 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.500+0000 7f408959a640 1 -- 172.21.15.146:0/153247849 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40840319d0 con 0x7f408c0b2400 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 -- 172.21.15.146:0/153247849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0b2400 msgr2=0x7f408c0b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 --2- 172.21.15.146:0/153247849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0b2400 0x7f408c0b2810 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f408c0d2650 tx=0x7f408402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 -- 172.21.15.146:0/153247849 shutdown_connections 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 --2- 172.21.15.146:0/153247849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0b2400 0x7f408c0b2810 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 --2- 172.21.15.146:0/153247849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0d2680 0x7f408c0d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 --2- 172.21.15.146:0/153247849 >> v2:172.21.15.177:3300/0 conn(0x7f408c0ccd40 0x7f408c09bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 -- 172.21.15.146:0/153247849 >> 172.21.15.146:0/153247849 conn(0x7f408c0b7400 msgr2=0x7f408c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 -- 172.21.15.146:0/153247849 shutdown_connections 2024-04-25T15:22:01.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.501+0000 7f4090a40640 1 -- 172.21.15.146:0/153247849 wait complete. 2024-04-25T15:22:01.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.503+0000 7f4090a40640 1 Processor -- start 2024-04-25T15:22:01.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.503+0000 7f4090a40640 1 -- start start 2024-04-25T15:22:01.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4090a40640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0b2400 0x7f408c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4090a40640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 0x7f408c15c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4090a40640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f408c0d2680 0x7f408c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4090a40640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f408c0b3940 con 0x7f408c0ccd40 2024-04-25T15:22:01.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4090a40640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f408c0b3fc0 con 0x7f408c0d2680 2024-04-25T15:22:01.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4090a40640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f408c0b3ac0 con 0x7f408c0b2400 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4089d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 0x7f408c15c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4089d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 0x7f408c15c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43540/0 (socket says 172.21.15.146:43540) 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f4089d9b640 1 -- 172.21.15.146:0/1162463372 learned_addr learned my addr 172.21.15.146:0/1162463372 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f408a59c640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0b2400 0x7f408c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.504+0000 7f408ad9d640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f408c0d2680 0x7f408c06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4089d9b640 1 -- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0b2400 msgr2=0x7f408c06ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4089d9b640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0b2400 0x7f408c06ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4089d9b640 1 -- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f408c0d2680 msgr2=0x7f408c06d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4089d9b640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f408c0d2680 0x7f408c06d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4089d9b640 1 -- 172.21.15.146:0/1162463372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c0c9c60 con 0x7f408c0ccd40 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4089d9b640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 0x7f408c15c490 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f408000d5f0 tx=0x7f408000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40800125f0 con 0x7f408c0ccd40 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4080019070 con 0x7f408c0ccd40 2024-04-25T15:22:01.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c0c9980 con 0x7f408c0ccd40 2024-04-25T15:22:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40800129d0 con 0x7f408c0ccd40 2024-04-25T15:22:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.505+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c0a0370 con 0x7f408c0ccd40 2024-04-25T15:22:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.506+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408c06a0e0 con 0x7f408c0ccd40 2024-04-25T15:22:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.507+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f4080012b70 con 0x7f408c0ccd40 2024-04-25T15:22:01.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.507+0000 7f407affd640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f406c074540 0x7f406c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.507+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f40800956c0 con 0x7f408c0ccd40 2024-04-25T15:22:01.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.507+0000 7f408a59c640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f406c074540 0x7f406c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.508+0000 7f408a59c640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f406c074540 0x7f406c076a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f408c0b8040 tx=0x7f407c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.512+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f408005f6a0 con 0x7f408c0ccd40 2024-04-25T15:22:01.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.687+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f408c0c9530 con 0x7f408c0ccd40 2024-04-25T15:22:01.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.688+0000 7f407affd640 1 -- 172.21.15.146:0/1162463372 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f40800639a0 con 0x7f408c0ccd40 2024-04-25T15:22:01.689 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:01.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f406c074540 msgr2=0x7f406c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f406c074540 0x7f406c076a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f408c0b8040 tx=0x7f407c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:01.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 msgr2=0x7f408c15c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 0x7f408c15c490 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f408000d5f0 tx=0x7f408000dab0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 shutdown_connections 2024-04-25T15:22:01.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f406c074540 0x7f406c076a50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f408c0d2680 0x7f408c06d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c0ccd40 0x7f408c15c490 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 --2- 172.21.15.146:0/1162463372 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f408c0b2400 0x7f408c06ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 >> 172.21.15.146:0/1162463372 conn(0x7f408c0b7400 msgr2=0x7f408c0afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:01.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 shutdown_connections 2024-04-25T15:22:01.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.690+0000 7f4090a40640 1 -- 172.21.15.146:0/1162463372 wait complete. 2024-04-25T15:22:01.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.813+0000 7fd0982e7640 1 Processor -- start 2024-04-25T15:22:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.813+0000 7fd0982e7640 1 -- start start 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd0982e7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd09006dc70 0x7fd0900b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd0982e7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0900b0520 con 0x7fd0900d10b0 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd0982e7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd090070d90 con 0x7fd090070500 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd0982e7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd0900b0850 con 0x7fd09006dc70 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd096884640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd0900d10b0 0x7fd0900b4610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52146/0 (socket says 172.21.15.146:52146) 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd096884640 1 -- 172.21.15.146:0/547214086 learned_addr learned my addr 172.21.15.146:0/547214086 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3803573950 0 0) 0x7fd0900b0520 con 0x7fd0900d10b0 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd07c003520 con 0x7fd0900d10b0 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd095882640 1 --2- 172.21.15.146:0/547214086 >> v2:172.21.15.177:3300/0 conn(0x7fd09006dc70 0x7fd0900b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 875290434 0 0) 0x7fd07c003520 con 0x7fd0900d10b0 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 >> v2:172.21.15.177:3300/0 conn(0x7fd09006dc70 msgr2=0x7fd0900b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.815+0000 7fd095081640 1 --2- 172.21.15.146:0/547214086 >> v2:172.21.15.177:3300/0 conn(0x7fd09006dc70 0x7fd0900b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 >> v1:172.21.15.177:6789/0 conn(0x7fd090070500 legacy=0x7fd09006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd090071070 con 0x7fd0900d10b0 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fd0880019f0 con 0x7fd0900d10b0 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 >> v1:172.21.15.146:6789/0 conn(0x7fd0900d10b0 legacy=0x7fd0900b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c0039b0 0x7fd07c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd07c004800 0x7fd07c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd07c005350 0x7fd07c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900b0850 con 0x7fd07c004800 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c00c230 con 0x7fd07c0039b0 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd07c00c430 con 0x7fd07c005350 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095882640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd07c004800 0x7fd07c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd096083640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c0039b0 0x7fd07c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd096884640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd07c005350 0x7fd07c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095882640 1 -- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd07c005350 msgr2=0x7fd07c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095882640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd07c005350 0x7fd07c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095882640 1 -- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c0039b0 msgr2=0x7fd07c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095882640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c0039b0 0x7fd07c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd095882640 1 -- 172.21.15.146:0/547214086 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c003520 con 0x7fd07c004800 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd096884640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd07c005350 0x7fd07c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.816+0000 7fd096884640 1 -- 172.21.15.146:0/547214086 reap_dead start 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.817+0000 7fd095882640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd07c004800 0x7fd07c004c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd08000ba30 tx=0x7fd07c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8477b4c18c1b97db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.817+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080030070 con 0x7fd07c004800 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.817+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08002ade0 con 0x7fd07c004800 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.817+0000 7fd095081640 1 -- 172.21.15.146:0/547214086 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08000b600 con 0x7fd07c004800 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 -- 172.21.15.146:0/547214086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd07c004800 msgr2=0x7fd07c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd07c004800 0x7fd07c004c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd08000ba30 tx=0x7fd07c005830 comp rx=0 tx=0).stop 2024-04-25T15:22:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 -- 172.21.15.146:0/547214086 shutdown_connections 2024-04-25T15:22:01.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 --2- 172.21.15.146:0/547214086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd07c004800 0x7fd07c004c10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 -- 172.21.15.146:0/547214086 >> 172.21.15.146:0/547214086 conn(0x7fd0900d1c10 msgr2=0x7fd0900733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:01.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 -- 172.21.15.146:0/547214086 shutdown_connections 2024-04-25T15:22:01.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.818+0000 7fd0982e7640 1 -- 172.21.15.146:0/547214086 wait complete. 2024-04-25T15:22:01.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.819+0000 7fd0982e7640 1 Processor -- start 2024-04-25T15:22:01.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.819+0000 7fd0982e7640 1 -- start start 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c004800 0x7fd090149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09015fb40 0x7fd09009ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 0x7fd09006f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900b0e60 con 0x7fd090147f90 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0900b0ce0 con 0x7fd07c004800 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd0900b0fe0 con 0x7fd09015fb40 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 0x7fd09006f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 0x7fd09006f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43568/0 (socket says 172.21.15.146:43568) 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 -- 172.21.15.146:0/2637235956 learned_addr learned my addr 172.21.15.146:0/2637235956 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096083640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c004800 0x7fd090149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd095882640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09015fb40 0x7fd09009ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 -- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09015fb40 msgr2=0x7fd09009ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09015fb40 0x7fd09009ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 -- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c004800 msgr2=0x7fd090149310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c004800 0x7fd090149310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 -- 172.21.15.146:0/2637235956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd090146ef0 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd096884640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 0x7fd09006f3f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd088002230 tx=0x7fd08800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800ed20 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd088018070 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0901471d0 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.821+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088013660 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.822+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd090147790 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.823+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd090146c40 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.823+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fd088013800 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.824+0000 7fd086ffd640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd074074570 0x7fd074076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.824+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd0880957b0 con 0x7fd090147f90 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.824+0000 7fd096083640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd074074570 0x7fd074076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:01.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.824+0000 7fd096083640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd074074570 0x7fd074076a80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd08c000c00 tx=0x7fd08c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:01.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:01.828+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd08805fde0 con 0x7fd090147f90 2024-04-25T15:22:02.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.007+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd09006a0e0 con 0x7fd090147f90 2024-04-25T15:22:02.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.007+0000 7fd086ffd640 1 -- 172.21.15.146:0/2637235956 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fd08805f6e0 con 0x7fd090147f90 2024-04-25T15:22:02.008 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:02.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd074074570 msgr2=0x7fd074076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd074074570 0x7fd074076a80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd08c000c00 tx=0x7fd08c009040 comp rx=0 tx=0).stop 2024-04-25T15:22:02.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 msgr2=0x7fd09006f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 0x7fd09006f3f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd088002230 tx=0x7fd08800e530 comp rx=0 tx=0).stop 2024-04-25T15:22:02.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 shutdown_connections 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd074074570 0x7fd074076a80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd090147f90 0x7fd09006f3f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09015fb40 0x7fd09009ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 --2- 172.21.15.146:0/2637235956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd07c004800 0x7fd090149310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 >> 172.21.15.146:0/2637235956 conn(0x7fd0900d1c10 msgr2=0x7fd0900b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 shutdown_connections 2024-04-25T15:22:02.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.009+0000 7fd0982e7640 1 -- 172.21.15.146:0/2637235956 wait complete. 2024-04-25T15:22:02.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:02.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.130+0000 7f5a1a6f9640 1 Processor -- start 2024-04-25T15:22:02.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.130+0000 7f5a1a6f9640 1 -- start start 2024-04-25T15:22:02.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a1a6f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b2df0 0x7f5a140d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a1a6f9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5a1408d370 0x7f5a140cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a1a6f9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a14160680 con 0x7f5a1406ee60 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a1a6f9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5a140d20e0 con 0x7f5a1408d370 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a1a6f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5a140d23c0 con 0x7f5a140b2df0 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a18c96640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5a1408d370 0x7f5a140cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a137fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5a1406ee60 0x7f5a1408cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52160/0 (socket says 172.21.15.146:52160) 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a137fe640 1 -- 172.21.15.146:0/2295460593 learned_addr learned my addr 172.21.15.146:0/2295460593 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.131+0000 7f5a13fff640 1 --2- 172.21.15.146:0/2295460593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b2df0 0x7f5a140d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a18c96640 1 -- 172.21.15.146:0/2295460593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b2df0 msgr2=0x7f5a140d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a18c96640 1 --2- 172.21.15.146:0/2295460593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b2df0 0x7f5a140d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a18c96640 1 -- 172.21.15.146:0/2295460593 >> v1:172.21.15.146:6789/0 conn(0x7f5a1406ee60 legacy=0x7f5a1408cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a18c96640 1 -- 172.21.15.146:0/2295460593 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a140d26a0 con 0x7f5a1408d370 2024-04-25T15:22:02.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a12ffd640 1 -- 172.21.15.146:0/2295460593 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208551792 0 0) 0x7f5a14160680 con 0x7f5a1406ee60 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a18c96640 1 --2- 172.21.15.146:0/2295460593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a1408d370 0x7f5a140cdec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009d90 tx=0x7f5a0c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9765bae5d20e1f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a12ffd640 1 -- 172.21.15.146:0/2295460593 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c031600 con 0x7f5a1408d370 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a12ffd640 1 -- 172.21.15.146:0/2295460593 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0317a0 con 0x7f5a1408d370 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.132+0000 7f5a12ffd640 1 -- 172.21.15.146:0/2295460593 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c031b70 con 0x7f5a1408d370 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/2295460593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a1408d370 msgr2=0x7f5a140cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/2295460593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a1408d370 0x7f5a140cdec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009d90 tx=0x7f5a0c02a5d0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/2295460593 shutdown_connections 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/2295460593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a1408d370 0x7f5a140cdec0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/2295460593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b2df0 0x7f5a140d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/2295460593 >> 172.21.15.146:0/2295460593 conn(0x7f5a140b5360 msgr2=0x7f5a140b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/2295460593 shutdown_connections 2024-04-25T15:22:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.133+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/2295460593 wait complete. 2024-04-25T15:22:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.135+0000 7f5a1a6f9640 1 Processor -- start 2024-04-25T15:22:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.135+0000 7f5a1a6f9640 1 -- start start 2024-04-25T15:22:02.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a1a6f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a1406ee60 0x7f5a14155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a1a6f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 0x7f5a1415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a1a6f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b3ca0 0x7f5a140b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a1a6f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a140b91e0 con 0x7f5a140b2df0 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a1a6f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5a140d2c60 con 0x7f5a1406ee60 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a1a6f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5a140d1c70 con 0x7f5a140b3ca0 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.136+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 0x7f5a1415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 0x7f5a1415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43590/0 (socket says 172.21.15.146:43590) 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 -- 172.21.15.146:0/963267516 learned_addr learned my addr 172.21.15.146:0/963267516 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a13fff640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a1406ee60 0x7f5a14155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a18c96640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b3ca0 0x7f5a140b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 -- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b3ca0 msgr2=0x7f5a140b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b3ca0 0x7f5a140b40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 -- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a1406ee60 msgr2=0x7f5a14155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a1406ee60 0x7f5a14155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 -- 172.21.15.146:0/963267516 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a140b45f0 con 0x7f5a140b2df0 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a18c96640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b3ca0 0x7f5a140b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:02.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a137fe640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 0x7f5a1415eac0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c3f0 tx=0x7f5a0800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a080074e0 con 0x7f5a140b2df0 2024-04-25T15:22:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a080047e0 con 0x7f5a140b2df0 2024-04-25T15:22:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a14162ed0 con 0x7f5a140b2df0 2024-04-25T15:22:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a08004c10 con 0x7f5a140b2df0 2024-04-25T15:22:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.137+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a14163490 con 0x7f5a140b2df0 2024-04-25T15:22:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.138+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a1406a0e0 con 0x7f5a140b2df0 2024-04-25T15:22:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.139+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f5a08011070 con 0x7f5a140b2df0 2024-04-25T15:22:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.139+0000 7f5a10ff9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5a00074570 0x7f5a00076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.139+0000 7f5a13fff640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5a00074570 0x7f5a00076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.139+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f5a08094ef0 con 0x7f5a140b2df0 2024-04-25T15:22:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.140+0000 7f5a13fff640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5a00074570 0x7f5a00076a80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5a1408cc00 tx=0x7f5a04009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:02.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.144+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a140b45f0 con 0x7f5a140b2df0 2024-04-25T15:22:02.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.330+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a140b8fc0 con 0x7f5a140b2df0 2024-04-25T15:22:02.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.330+0000 7f5a10ff9640 1 -- 172.21.15.146:0/963267516 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f5a0805f520 con 0x7f5a140b2df0 2024-04-25T15:22:02.331 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6835/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:22:02.331 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.177:6833/539736758 2024-04-26T15:21:49.486202+0000 2024-04-25T15:22:02.332 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:22:02.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5a00074570 msgr2=0x7f5a00076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5a00074570 0x7f5a00076a80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5a1408cc00 tx=0x7f5a04009110 comp rx=0 tx=0).stop 2024-04-25T15:22:02.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 msgr2=0x7f5a1415eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 0x7f5a1415eac0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c3f0 tx=0x7f5a0800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 shutdown_connections 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5a00074570 0x7f5a00076a80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a140b3ca0 0x7f5a140b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a140b2df0 0x7f5a1415eac0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 --2- 172.21.15.146:0/963267516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a1406ee60 0x7f5a14155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 >> 172.21.15.146:0/963267516 conn(0x7f5a140b5360 msgr2=0x7f5a140d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 shutdown_connections 2024-04-25T15:22:02.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.332+0000 7f5a1a6f9640 1 -- 172.21.15.146:0/963267516 wait complete. 2024-04-25T15:22:02.344 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:22:02.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:02.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-25T15:22:02.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[42921]: fuse finished with error 0 and tester_r 0 2024-04-25T15:22:02.374 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:22:02.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:02.375 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T15:22:02.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.655+0000 7f5a0ee61640 1 Processor -- start 2024-04-25T15:22:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.655+0000 7f5a0ee61640 1 -- start start 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f5a0ee61640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f5a0808d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f5a0ee61640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a080ce4c0 con 0x7f5a080d21f0 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f5a0ee61640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a0808d5b0 con 0x7f5a080b8de0 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f5a0ee61640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5a080d2a20 con 0x7f5a0808cc60 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59fffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5a080d21f0 0x7f5a080cf4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52174/0 (socket says 172.21.15.146:52174) 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59fffff640 1 -- 172.21.15.146:0/1951885038 learned_addr learned my addr 172.21.15.146:0/1951885038 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f5a0d3fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f5a0808d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778873674 0 0) 0x7f59f8001110 con 0x7f5a080d21f0 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f0003520 con 0x7f5a080d21f0 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3556215701 0 0) 0x7f5a04001110 con 0x7f5a080b8de0 2024-04-25T15:22:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f8001110 con 0x7f5a080b8de0 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3675445587 0 0) 0x7f59f0003520 con 0x7f5a080d21f0 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 msgr2=0x7f5a0808d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f5a0808d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 >> v1:172.21.15.177:6789/0 conn(0x7f5a080b8de0 legacy=0x7f5a080b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a080ce740 con 0x7f5a080d21f0 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f59f8001840 con 0x7f5a080d21f0 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f5a0d3fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f5a0808d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 >> v1:172.21.15.146:6789/0 conn(0x7f5a080d21f0 legacy=0x7f5a080cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f59f0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f59f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f0004d40 0x7f59f0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a080d2a20 con 0x7f59f0003b70 2024-04-25T15:22:02.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59f000baa0 con 0x7f59f0004d40 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.657+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f59f000bca0 con 0x7f5a0808cc60 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f59f0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f59fffff640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f59f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0d3fe640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f0004d40 0x7f59f0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 msgr2=0x7f59f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a0808cc60 0x7f59f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f0004d40 msgr2=0x7f59f0005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f0004d40 0x7f59f0005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/1951885038 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f0003520 con 0x7f59f0003b70 2024-04-25T15:22:02.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.658+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f59f0003fc0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5a04001a70 tx=0x7f59f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c06ae45651e4bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040016c0 con 0x7f59f0003b70 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0402ae10 con 0x7f59f0003b70 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f59ff7fe640 1 -- 172.21.15.146:0/1951885038 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040052f0 con 0x7f59f0003b70 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0ee61640 1 -- 172.21.15.146:0/1951885038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 msgr2=0x7f59f0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/1951885038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f59f0003fc0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5a04001a70 tx=0x7f59f0005830 comp rx=0 tx=0).stop 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0d3fe640 1 -- 172.21.15.146:0/1951885038 reap_dead start 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0ee61640 1 -- 172.21.15.146:0/1951885038 shutdown_connections 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0ee61640 1 -- 172.21.15.146:0/1951885038 >> 172.21.15.146:0/1951885038 conn(0x7f5a080d0310 msgr2=0x7f5a080d0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0ee61640 1 -- 172.21.15.146:0/1951885038 shutdown_connections 2024-04-25T15:22:02.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.659+0000 7f5a0ee61640 1 -- 172.21.15.146:0/1951885038 wait complete. 2024-04-25T15:22:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.661+0000 7f5a0ee61640 1 Processor -- start 2024-04-25T15:22:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.661+0000 7f5a0ee61640 1 -- start start 2024-04-25T15:22:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f5a08149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a08156600 0x7f5a0814ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a081491f0 0x7f5a08149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a080cefc0 con 0x7f59f0003b70 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5a080cece0 con 0x7f5a08156600 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5a080d2600 con 0x7f5a081491f0 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f5a08149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f5a08149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43620/0 (socket says 172.21.15.146:43620) 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/3155945089 learned_addr learned my addr 172.21.15.146:0/3155945089 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0d3fe640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a081491f0 0x7f5a08149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f59fffff640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a08156600 0x7f5a0814ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a081491f0 msgr2=0x7f5a08149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a081491f0 0x7f5a08149600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a08156600 msgr2=0x7f5a0814ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a08156600 0x7f5a0814ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 -- 172.21.15.146:0/3155945089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a08149b40 con 0x7f59f0003b70 2024-04-25T15:22:02.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0d3fe640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a081491f0 0x7f5a08149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f59fffff640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a08156600 0x7f5a0814ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0cbfd640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f5a08149db0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5a04005040 tx=0x7f5a040014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04042070 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a04006e50 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a081615e0 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04037040 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.663+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a08158f00 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.665+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0806a0e0 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.665+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f5a04045070 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.666+0000 7f59fd7fa640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f59f4074570 0x7f59f4076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.666+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f5a040b4eb0 con 0x7f59f0003b70 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.666+0000 7f59fffff640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f59f4074570 0x7f59f4076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:02.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.666+0000 7f59fffff640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f59f4074570 0x7f59f4076a80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59f8000c00 tx=0x7f59f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:02.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.670+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a08149b40 con 0x7f59f0003b70 2024-04-25T15:22:02.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.882+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a0815b550 con 0x7f59f0003b70 2024-04-25T15:22:02.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.883+0000 7f59fd7fa640 1 -- 172.21.15.146:0/3155945089 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5192 (secure 0 0 0) 0x7f5a0808d5b0 con 0x7f59f0003b70 2024-04-25T15:22:02.884 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:02.884 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:53.503394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5553,"qdb_cluster":[5553]},"id":2}]} 2024-04-25T15:22:02.884 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f59f4074570 msgr2=0x7f59f4076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f59f4074570 0x7f59f4076a80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59f8000c00 tx=0x7f59f8009040 comp rx=0 tx=0).stop 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 msgr2=0x7f5a08149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f5a08149db0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5a04005040 tx=0x7f5a040014a0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 shutdown_connections 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f59f4074570 0x7f59f4076a80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5a081491f0 0x7f5a08149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5a08156600 0x7f5a0814ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 --2- 172.21.15.146:0/3155945089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59f0003b70 0x7f5a08149db0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:02.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 >> 172.21.15.146:0/3155945089 conn(0x7f5a080d0310 msgr2=0x7f5a080686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:02.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 shutdown_connections 2024-04-25T15:22:02.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:02.885+0000 7f5a0ee61640 1 -- 172.21.15.146:0/3155945089 wait complete. 2024-04-25T15:22:02.897 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:03.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.004+0000 7f5336ef9640 1 Processor -- start 2024-04-25T15:22:03.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.004+0000 7f5336ef9640 1 -- start start 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.005+0000 7f5336ef9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53300d4f60 0x7f53300d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.005+0000 7f5336ef9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53300b9200 con 0x7f533006ccc0 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.005+0000 7f5336ef9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53300d1070 con 0x7f53300d0020 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.005+0000 7f5336ef9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f533006c550 con 0x7f53300d4f60 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f5334c95640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53300d4f60 0x7f53300d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f5335496640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f533006ccc0 0x7f533006d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52188/0 (socket says 172.21.15.146:52188) 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f5335496640 1 -- 172.21.15.146:0/1234348859 learned_addr learned my addr 172.21.15.146:0/1234348859 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f5334c95640 1 -- 172.21.15.146:0/1234348859 >> v1:172.21.15.177:6789/0 conn(0x7f53300d0020 legacy=0x7f53300d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f5334c95640 1 -- 172.21.15.146:0/1234348859 >> v1:172.21.15.146:6789/0 conn(0x7f533006ccc0 legacy=0x7f533006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f5334c95640 1 -- 172.21.15.146:0/1234348859 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53300d1310 con 0x7f53300d4f60 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.006+0000 7f532f7fe640 1 -- 172.21.15.146:0/1234348859 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426960551 0 0) 0x7f5328001490 con 0x7f533006ccc0 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5334c95640 1 --2- 172.21.15.146:0/1234348859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53300d4f60 0x7f53300d5370 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f53240097a0 tx=0x7f532402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21cf01a2b3e6f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f532f7fe640 1 -- 172.21.15.146:0/1234348859 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5324037070 con 0x7f53300d4f60 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f532f7fe640 1 -- 172.21.15.146:0/1234348859 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5324032440 con 0x7f53300d4f60 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f532f7fe640 1 -- 172.21.15.146:0/1234348859 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532402ae60 con 0x7f53300d4f60 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 -- 172.21.15.146:0/1234348859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53300d4f60 msgr2=0x7f53300d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1234348859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53300d4f60 0x7f53300d5370 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f53240097a0 tx=0x7f532402a110 comp rx=0 tx=0).stop 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 -- 172.21.15.146:0/1234348859 shutdown_connections 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1234348859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53300d4f60 0x7f53300d5370 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 -- 172.21.15.146:0/1234348859 >> 172.21.15.146:0/1234348859 conn(0x7f53300d2450 msgr2=0x7f53300d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 -- 172.21.15.146:0/1234348859 shutdown_connections 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.007+0000 7f5336ef9640 1 -- 172.21.15.146:0/1234348859 wait complete. 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.009+0000 7f5336ef9640 1 Processor -- start 2024-04-25T15:22:03.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.009+0000 7f5336ef9640 1 -- start start 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 0x7f53301ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53300d0020 0x7f53301e5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53300d4f60 0x7f53301e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f53300b5150 con 0x7f533006ccc0 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f533006c830 con 0x7f53300d0020 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f53300b52d0 con 0x7f53300d4f60 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 0x7f53301ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 0x7f53301ea410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43646/0 (socket says 172.21.15.146:43646) 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 -- 172.21.15.146:0/1658146669 learned_addr learned my addr 172.21.15.146:0/1658146669 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 -- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53300d4f60 msgr2=0x7f53301e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53300d4f60 0x7f53301e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 -- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53300d0020 msgr2=0x7f53301e5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f532ffff640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53300d0020 0x7f53301e5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53300d0020 0x7f53301e5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 -- 172.21.15.146:0/1658146669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533015fdc0 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5334c95640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 0x7f53301ea410 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f53240098c0 tx=0x7f53240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5324037040 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5324042070 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533015d460 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5324030a70 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.011+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f533015da20 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.012+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f533006a0e0 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.013+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f5324030c10 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.013+0000 7f532d7fa640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5314074540 0x7f5314076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.013+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f53240b5e90 con 0x7f533006ccc0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.013+0000 7f532ffff640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5314074540 0x7f5314076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.014+0000 7f532ffff640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5314074540 0x7f5314076a50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f533015c3d0 tx=0x7f531c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.018+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53240804c0 con 0x7f533006ccc0 2024-04-25T15:22:03.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.194+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53300badc0 con 0x7f533006ccc0 2024-04-25T15:22:03.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.195+0000 7f532d7fa640 1 -- 172.21.15.146:0/1658146669 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7f532407fdc0 con 0x7f533006ccc0 2024-04-25T15:22:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:03.196 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:03.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5314074540 msgr2=0x7f5314076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5314074540 0x7f5314076a50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f533015c3d0 tx=0x7f531c009040 comp rx=0 tx=0).stop 2024-04-25T15:22:03.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 msgr2=0x7f53301ea410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 0x7f53301ea410 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f53240098c0 tx=0x7f53240046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 shutdown_connections 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5314074540 0x7f5314076a50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53300d4f60 0x7f53301e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53300d0020 0x7f53301e5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 --2- 172.21.15.146:0/1658146669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f533006ccc0 0x7f53301ea410 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 >> 172.21.15.146:0/1658146669 conn(0x7f53300d2450 msgr2=0x7f53300d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.197+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 shutdown_connections 2024-04-25T15:22:03.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.198+0000 7f5336ef9640 1 -- 172.21.15.146:0/1658146669 wait complete. 2024-04-25T15:22:03.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:21:06.188796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:21:06.574527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:03.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:03.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.322+0000 7facdeebd640 1 Processor -- start 2024-04-25T15:22:03.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.322+0000 7facdeebd640 1 -- start start 2024-04-25T15:22:03.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdeebd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7facd80b2cd0 0x7facd80cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdeebd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d1920 0x7facd80d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdeebd640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd80cf7d0 con 0x7facd80d2270 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdeebd640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7facd80d0070 con 0x7facd80b2cd0 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdeebd640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7facd80717e0 con 0x7facd80d1920 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdcc59640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7facd80b2cd0 0x7facd80cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdd45a640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7facd80d2270 0x7facd808ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:55432/0 (socket says 172.21.15.146:55432) 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facdd45a640 1 -- 172.21.15.146:0/3588372159 learned_addr learned my addr 172.21.15.146:0/3588372159 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.323+0000 7facd7fff640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d1920 0x7facd80d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.324+0000 7facdcc59640 1 -- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d1920 msgr2=0x7facd80d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.324+0000 7facdcc59640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d1920 0x7facd80d1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.324+0000 7facdcc59640 1 -- 172.21.15.146:0/3588372159 >> v1:172.21.15.177:6789/0 conn(0x7facd80d2270 legacy=0x7facd808ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.324+0000 7facdcc59640 1 -- 172.21.15.146:0/3588372159 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd80cfab0 con 0x7facd80b2cd0 2024-04-25T15:22:03.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.324+0000 7facd7fff640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d1920 0x7facd80d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:03.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.324+0000 7facdcc59640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80b2cd0 0x7facd80cf0f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7facc8009790 tx=0x7facc802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27c7f9e1d4ad4bc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facd77fe640 1 -- 172.21.15.146:0/3588372159 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8037070 con 0x7facd80b2cd0 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facd77fe640 1 -- 172.21.15.146:0/3588372159 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc802ae50 con 0x7facd80b2cd0 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facd77fe640 1 -- 172.21.15.146:0/3588372159 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8032660 con 0x7facd80b2cd0 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 -- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80b2cd0 msgr2=0x7facd80cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80b2cd0 0x7facd80cf0f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7facc8009790 tx=0x7facc802a100 comp rx=0 tx=0).stop 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 -- 172.21.15.146:0/3588372159 shutdown_connections 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d1920 0x7facd80d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 --2- 172.21.15.146:0/3588372159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80b2cd0 0x7facd80cf0f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 -- 172.21.15.146:0/3588372159 >> 172.21.15.146:0/3588372159 conn(0x7facd80b52e0 msgr2=0x7facd80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 -- 172.21.15.146:0/3588372159 shutdown_connections 2024-04-25T15:22:03.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.325+0000 7facdeebd640 1 -- 172.21.15.146:0/3588372159 wait complete. 2024-04-25T15:22:03.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.327+0000 7facdeebd640 1 Processor -- start 2024-04-25T15:22:03.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.327+0000 7facdeebd640 1 -- start start 2024-04-25T15:22:03.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 0x7facd8159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80d1920 0x7facd80b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d2270 0x7facd80b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7facd8072c80 con 0x7facd80b2cd0 2024-04-25T15:22:03.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7facd80d2680 con 0x7facd80d1920 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7facd8072e00 con 0x7facd80d2270 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 0x7facd8159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 0x7facd8159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43656/0 (socket says 172.21.15.146:43656) 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 -- 172.21.15.146:0/145265997 learned_addr learned my addr 172.21.15.146:0/145265997 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facd7fff640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80d1920 0x7facd80b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdd45a640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d2270 0x7facd80b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 -- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d2270 msgr2=0x7facd80b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d2270 0x7facd80b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 -- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80d1920 msgr2=0x7facd80b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80d1920 0x7facd80b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 -- 172.21.15.146:0/145265997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd80cceb0 con 0x7facd80b2cd0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdcc59640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 0x7facd8159e40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7facc8035040 tx=0x7facc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8041020 con 0x7facd80b2cd0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc802ac50 con 0x7facd80b2cd0 2024-04-25T15:22:03.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.329+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd80cd3f0 con 0x7facd80b2cd0 2024-04-25T15:22:03.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.330+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8037040 con 0x7facd80b2cd0 2024-04-25T15:22:03.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.330+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd80cd9b0 con 0x7facd80b2cd0 2024-04-25T15:22:03.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.330+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd806a0e0 con 0x7facd80b2cd0 2024-04-25T15:22:03.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.331+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7facc8044020 con 0x7facd80b2cd0 2024-04-25T15:22:03.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.332+0000 7facd57fa640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facc4074570 0x7facc4076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.332+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7facc80b55a0 con 0x7facd80b2cd0 2024-04-25T15:22:03.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.332+0000 7facd7fff640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facc4074570 0x7facc4076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.332+0000 7facd7fff640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facc4074570 0x7facc4076a80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faccc009560 tx=0x7faccc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.336+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7facc802ae20 con 0x7facd80b2cd0 2024-04-25T15:22:03.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.511+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7facd815f750 con 0x7facd80b2cd0 2024-04-25T15:22:03.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.512+0000 7facd57fa640 1 -- 172.21.15.146:0/145265997 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7facc804b030 con 0x7facd80b2cd0 2024-04-25T15:22:03.513 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facc4074570 msgr2=0x7facc4076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facc4074570 0x7facc4076a80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faccc009560 tx=0x7faccc009190 comp rx=0 tx=0).stop 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 msgr2=0x7facd8159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 0x7facd8159e40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7facc8035040 tx=0x7facc80046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 shutdown_connections 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facc4074570 0x7facc4076a80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd80d2270 0x7facd80b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd80d1920 0x7facd80b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 --2- 172.21.15.146:0/145265997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd80b2cd0 0x7facd8159e40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 >> 172.21.15.146:0/145265997 conn(0x7facd80b52e0 msgr2=0x7facd80b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 shutdown_connections 2024-04-25T15:22:03.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.514+0000 7facdeebd640 1 -- 172.21.15.146:0/145265997 wait complete. 2024-04-25T15:22:03.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:03.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.637+0000 7f229e230640 1 Processor -- start 2024-04-25T15:22:03.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.637+0000 7f229e230640 1 -- start start 2024-04-25T15:22:03.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f229e230640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980b2cd0 0x7f22980cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f229e230640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f22980d2440 0x7f2298071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f229e230640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22980cf7d0 con 0x7f22980d1920 2024-04-25T15:22:03.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f229e230640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f229808da90 con 0x7f22980d2440 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f229e230640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f22980d0070 con 0x7f22980b2cd0 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f2296ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f22980d1920 0x7f22980d1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52190/0 (socket says 172.21.15.146:52190) 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f2296ffd640 1 -- 172.21.15.146:0/2954015204 learned_addr learned my addr 172.21.15.146:0/2954015204 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f2297fff640 1 --2- 172.21.15.146:0/2954015204 >> v2:172.21.15.177:3300/0 conn(0x7f22980d2440 0x7f2298071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22977fe640 1 --2- 172.21.15.146:0/2954015204 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980b2cd0 0x7f22980cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22967fc640 1 -- 172.21.15.146:0/2954015204 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 307910104 0 0) 0x7f22980cf7d0 con 0x7f22980d1920 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22967fc640 1 -- 172.21.15.146:0/2954015204 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2280003520 con 0x7f22980d1920 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22977fe640 1 -- 172.21.15.146:0/2954015204 >> v2:172.21.15.177:3300/0 conn(0x7f22980d2440 msgr2=0x7f2298071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22977fe640 1 --2- 172.21.15.146:0/2954015204 >> v2:172.21.15.177:3300/0 conn(0x7f22980d2440 0x7f2298071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22977fe640 1 -- 172.21.15.146:0/2954015204 >> v1:172.21.15.146:6789/0 conn(0x7f22980d1920 legacy=0x7f22980d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f22977fe640 1 -- 172.21.15.146:0/2954015204 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22980cfab0 con 0x7f22980b2cd0 2024-04-25T15:22:03.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.639+0000 7f2297fff640 1 --2- 172.21.15.146:0/2954015204 >> v2:172.21.15.177:3300/0 conn(0x7f22980d2440 0x7f2298071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.640+0000 7f22977fe640 1 --2- 172.21.15.146:0/2954015204 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980b2cd0 0x7f22980cf0f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2288009790 tx=0x7f228802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1bf7be03dc021da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.640+0000 7f22967fc640 1 -- 172.21.15.146:0/2954015204 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288037070 con 0x7f22980b2cd0 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.640+0000 7f22967fc640 1 -- 172.21.15.146:0/2954015204 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228802ae50 con 0x7f22980b2cd0 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.640+0000 7f22967fc640 1 -- 172.21.15.146:0/2954015204 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288032640 con 0x7f22980b2cd0 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 -- 172.21.15.146:0/2954015204 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980b2cd0 msgr2=0x7f22980cf0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 --2- 172.21.15.146:0/2954015204 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980b2cd0 0x7f22980cf0f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2288009790 tx=0x7f228802a100 comp rx=0 tx=0).stop 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 -- 172.21.15.146:0/2954015204 shutdown_connections 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 --2- 172.21.15.146:0/2954015204 >> v2:172.21.15.177:3300/0 conn(0x7f22980d2440 0x7f2298071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 --2- 172.21.15.146:0/2954015204 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980b2cd0 0x7f22980cf0f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 -- 172.21.15.146:0/2954015204 >> 172.21.15.146:0/2954015204 conn(0x7f22980b52e0 msgr2=0x7f22980b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 -- 172.21.15.146:0/2954015204 shutdown_connections 2024-04-25T15:22:03.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.641+0000 7f229e230640 1 -- 172.21.15.146:0/2954015204 wait complete. 2024-04-25T15:22:03.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.643+0000 7f229e230640 1 Processor -- start 2024-04-25T15:22:03.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.643+0000 7f229e230640 1 -- start start 2024-04-25T15:22:03.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f229e230640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22980b2cd0 0x7f229814b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f229e230640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 0x7f22981e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f229e230640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980d2440 0x7f22981f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f229e230640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f22980721b0 con 0x7f22980d1920 2024-04-25T15:22:03.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f229e230640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22980717e0 con 0x7f22980b2cd0 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f229e230640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2298072330 con 0x7f22980d2440 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f2296ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 0x7f22981e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f2296ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 0x7f22981e67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43668/0 (socket says 172.21.15.146:43668) 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.644+0000 7f2296ffd640 1 -- 172.21.15.146:0/1547542958 learned_addr learned my addr 172.21.15.146:0/1547542958 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f22977fe640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22980b2cd0 0x7f229814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2297fff640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980d2440 0x7f22981f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2296ffd640 1 -- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980d2440 msgr2=0x7f22981f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2296ffd640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980d2440 0x7f22981f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2296ffd640 1 -- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22980b2cd0 msgr2=0x7f229814b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2296ffd640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22980b2cd0 0x7f229814b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2296ffd640 1 -- 172.21.15.146:0/1547542958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22981f4090 con 0x7f22980d1920 2024-04-25T15:22:03.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f2296ffd640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 0x7f22981e67e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f228c00c390 tx=0x7f228c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c017070 con 0x7f22980d1920 2024-04-25T15:22:03.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228c005ce0 con 0x7f22980d1920 2024-04-25T15:22:03.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229814b950 con 0x7f22980d1920 2024-04-25T15:22:03.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c0072f0 con 0x7f22980d1920 2024-04-25T15:22:03.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.645+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229814bf10 con 0x7f22980d1920 2024-04-25T15:22:03.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.646+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f229806a0e0 con 0x7f22980d1920 2024-04-25T15:22:03.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.647+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f228c01a070 con 0x7f22980d1920 2024-04-25T15:22:03.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.647+0000 7f227ffff640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2284074570 0x7f2284076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.647+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f228c0946f0 con 0x7f22980d1920 2024-04-25T15:22:03.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.647+0000 7f22977fe640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2284074570 0x7f2284076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.648+0000 7f22977fe640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2284074570 0x7f2284076a80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2288034000 tx=0x7f228802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.652+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f228c05e5c0 con 0x7f22980d1920 2024-04-25T15:22:03.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.825+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f229814b6e0 con 0x7f22980d1920 2024-04-25T15:22:03.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.825+0000 7f227ffff640 1 -- 172.21.15.146:0/1547542958 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f228c0628c0 con 0x7f22980d1920 2024-04-25T15:22:03.826 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:03.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2284074570 msgr2=0x7f2284076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2284074570 0x7f2284076a80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2288034000 tx=0x7f228802e000 comp rx=0 tx=0).stop 2024-04-25T15:22:03.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 msgr2=0x7f22981e67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 0x7f22981e67e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f228c00c390 tx=0x7f228c00c790 comp rx=0 tx=0).stop 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 shutdown_connections 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2284074570 0x7f2284076a80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22980d2440 0x7f22981f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22980d1920 0x7f22981e67e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 --2- 172.21.15.146:0/1547542958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22980b2cd0 0x7f229814b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.827+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 >> 172.21.15.146:0/1547542958 conn(0x7f22980b52e0 msgr2=0x7f22980d0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.828+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 shutdown_connections 2024-04-25T15:22:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.828+0000 7f229e230640 1 -- 172.21.15.146:0/1547542958 wait complete. 2024-04-25T15:22:03.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:03.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.949+0000 7fd594d9f640 1 Processor -- start 2024-04-25T15:22:03.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.949+0000 7fd594d9f640 1 -- start start 2024-04-25T15:22:03.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.950+0000 7fd594d9f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd590160170 0x7fd59016aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.950+0000 7fd594d9f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900d5150 con 0x7fd59006c550 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.950+0000 7fd594d9f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59006cb30 con 0x7fd59006dbd0 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.950+0000 7fd594d9f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd59006ce10 con 0x7fd590160170 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd590160170 0x7fd59016aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58effd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd59006c550 0x7fd5900d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52192/0 (socket says 172.21.15.146:52192) 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58effd640 1 -- 172.21.15.146:0/1641373414 learned_addr learned my addr 172.21.15.146:0/1641373414 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58ffff640 1 -- 172.21.15.146:0/1641373414 >> v1:172.21.15.177:6789/0 conn(0x7fd59006dbd0 legacy=0x7fd5900d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58ffff640 1 -- 172.21.15.146:0/1641373414 >> v1:172.21.15.146:6789/0 conn(0x7fd59006c550 legacy=0x7fd5900d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58ffff640 1 -- 172.21.15.146:0/1641373414 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59006d0f0 con 0x7fd590160170 2024-04-25T15:22:03.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.951+0000 7fd58e7fc640 1 -- 172.21.15.146:0/1641373414 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327619012 0 0) 0x7fd5900d5150 con 0x7fd59006c550 2024-04-25T15:22:03.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.952+0000 7fd58ffff640 1 --2- 172.21.15.146:0/1641373414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd590160170 0x7fd59016aee0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd588009d90 tx=0x7fd58802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b53ecb7989a6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.952+0000 7fd58e7fc640 1 -- 172.21.15.146:0/1641373414 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588007740 con 0x7fd590160170 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.952+0000 7fd58e7fc640 1 -- 172.21.15.146:0/1641373414 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5880078e0 con 0x7fd590160170 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.952+0000 7fd58e7fc640 1 -- 172.21.15.146:0/1641373414 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588007cb0 con 0x7fd590160170 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.952+0000 7fd594d9f640 1 -- 172.21.15.146:0/1641373414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd590160170 msgr2=0x7fd59016aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.952+0000 7fd594d9f640 1 --2- 172.21.15.146:0/1641373414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd590160170 0x7fd59016aee0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd588009d90 tx=0x7fd58802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.953+0000 7fd594d9f640 1 -- 172.21.15.146:0/1641373414 shutdown_connections 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.953+0000 7fd594d9f640 1 --2- 172.21.15.146:0/1641373414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd590160170 0x7fd59016aee0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.953+0000 7fd594d9f640 1 -- 172.21.15.146:0/1641373414 >> 172.21.15.146:0/1641373414 conn(0x7fd5900d6400 msgr2=0x7fd5900b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.953+0000 7fd594d9f640 1 -- 172.21.15.146:0/1641373414 shutdown_connections 2024-04-25T15:22:03.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.953+0000 7fd594d9f640 1 -- 172.21.15.146:0/1641373414 wait complete. 2024-04-25T15:22:03.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.954+0000 7fd594d9f640 1 Processor -- start 2024-04-25T15:22:03.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.954+0000 7fd594d9f640 1 -- start start 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd594d9f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 0x7fd590150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd594d9f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd59006dbd0 0x7fd590156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd594d9f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd590160170 0x7fd5901574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd594d9f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd5900cc5f0 con 0x7fd59006c550 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd594d9f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd59006d6b0 con 0x7fd59006dbd0 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd594d9f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd59016b420 con 0x7fd590160170 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 0x7fd590150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 0x7fd590150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43682/0 (socket says 172.21.15.146:43682) 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 -- 172.21.15.146:0/82880863 learned_addr learned my addr 172.21.15.146:0/82880863 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58ffff640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd590160170 0x7fd5901574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58effd640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd59006dbd0 0x7fd590156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 -- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd590160170 msgr2=0x7fd5901574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd590160170 0x7fd5901574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 -- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd59006dbd0 msgr2=0x7fd590156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd59006dbd0 0x7fd590156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 -- 172.21.15.146:0/82880863 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5900cd5d0 con 0x7fd59006c550 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58ffff640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd590160170 0x7fd5901574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58effd640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd59006dbd0 0x7fd590156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.956+0000 7fd58f7fe640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 0x7fd590150f70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd58000b420 tx=0x7fd58000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.957+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580011830 con 0x7fd59006c550 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.957+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd580016070 con 0x7fd59006c550 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.957+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5900ce400 con 0x7fd59006c550 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.957+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580011c10 con 0x7fd59006c550 2024-04-25T15:22:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.957+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5900ccaf0 con 0x7fd59006c550 2024-04-25T15:22:03.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.958+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd590050740 con 0x7fd59006c550 2024-04-25T15:22:03.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.958+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fd58001b460 con 0x7fd59006c550 2024-04-25T15:22:03.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.959+0000 7fd577fff640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd57c074570 0x7fd57c076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:03.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.959+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd580095090 con 0x7fd59006c550 2024-04-25T15:22:03.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.959+0000 7fd58effd640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd57c074570 0x7fd57c076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:03.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.959+0000 7fd58effd640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd57c074570 0x7fd57c076a80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd59016b3f0 tx=0x7fd584009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:03.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:03.963+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd58005f6c0 con 0x7fd59006c550 2024-04-25T15:22:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.138+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd5900d3c00 con 0x7fd59006c550 2024-04-25T15:22:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.139+0000 7fd577fff640 1 -- 172.21.15.146:0/82880863 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fd58005efc0 con 0x7fd59006c550 2024-04-25T15:22:04.139 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd57c074570 msgr2=0x7fd57c076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd57c074570 0x7fd57c076a80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd59016b3f0 tx=0x7fd584009040 comp rx=0 tx=0).stop 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 msgr2=0x7fd590150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 0x7fd590150f70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd58000b420 tx=0x7fd58000b8e0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 shutdown_connections 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd57c074570 0x7fd57c076a80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd590160170 0x7fd5901574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd59006dbd0 0x7fd590156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.140+0000 7fd594d9f640 1 --2- 172.21.15.146:0/82880863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd59006c550 0x7fd590150f70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.141+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 >> 172.21.15.146:0/82880863 conn(0x7fd5900d6400 msgr2=0x7fd5900b9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.141+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 shutdown_connections 2024-04-25T15:22:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.141+0000 7fd594d9f640 1 -- 172.21.15.146:0/82880863 wait complete. 2024-04-25T15:22:04.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:04.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.259+0000 7f18eb684640 1 Processor -- start 2024-04-25T15:22:04.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.259+0000 7f18eb684640 1 -- start start 2024-04-25T15:22:04.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18eb684640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18e4070360 0x7f18e416acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18eb684640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18e4071bc0 con 0x7f18e40b7350 2024-04-25T15:22:04.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18eb684640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18e40d1b00 con 0x7f18e40715e0 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18eb684640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f18e40d1de0 con 0x7f18e4070360 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18e9420640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f18e40b7350 0x7f18e4160000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52200/0 (socket says 172.21.15.146:52200) 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18e9420640 1 -- 172.21.15.146:0/871159377 learned_addr learned my addr 172.21.15.146:0/871159377 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413877049 0 0) 0x7f18e4071bc0 con 0x7f18e40b7350 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18d0003520 con 0x7f18e40b7350 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18e9c21640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18e4070360 0x7f18e416acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2440629110 0 0) 0x7f18d0003520 con 0x7f18e40b7350 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18e4070360 msgr2=0x7f18e416acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18e4070360 0x7f18e416acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 >> v1:172.21.15.177:6789/0 conn(0x7f18e40715e0 legacy=0x7f18e406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e40d20c0 con 0x7f18e40b7350 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f18e0001670 con 0x7f18e40b7350 2024-04-25T15:22:04.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 >> v1:172.21.15.146:6789/0 conn(0x7f18e40b7350 legacy=0x7f18e4160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d00039e0 0x7f18d0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d0004800 0x7f18d0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18d0005150 0x7f18d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f18e40d1de0 con 0x7f18d0004800 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18d000bfb0 con 0x7f18d0005150 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f18d000c1b0 con 0x7f18d00039e0 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.261+0000 7f18e8c1f640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d0004800 0x7f18d0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e9420640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d00039e0 0x7f18d0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e9c21640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18d0005150 0x7f18d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e8c1f640 1 -- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d00039e0 msgr2=0x7f18d0003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e8c1f640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d00039e0 0x7f18d0003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e8c1f640 1 -- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18d0005150 msgr2=0x7f18d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e8c1f640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18d0005150 0x7f18d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e8c1f640 1 -- 172.21.15.146:0/871159377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d0003520 con 0x7f18d0004800 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.262+0000 7f18e8c1f640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d0004800 0x7f18d0004c10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f18d4001750 tx=0x7f18d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb7bffe125ed862 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4001370 con 0x7f18d0004800 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18d40052d0 con 0x7f18d0004800 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18dbfff640 1 -- 172.21.15.146:0/871159377 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d402abe0 con 0x7f18d0004800 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18eb684640 1 -- 172.21.15.146:0/871159377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d0004800 msgr2=0x7f18d0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18eb684640 1 --2- 172.21.15.146:0/871159377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d0004800 0x7f18d0004c10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f18d4001750 tx=0x7f18d0005830 comp rx=0 tx=0).stop 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18e9c21640 1 -- 172.21.15.146:0/871159377 reap_dead start 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18eb684640 1 -- 172.21.15.146:0/871159377 shutdown_connections 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18eb684640 1 -- 172.21.15.146:0/871159377 >> 172.21.15.146:0/871159377 conn(0x7f18e409b2b0 msgr2=0x7f18e40b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18eb684640 1 -- 172.21.15.146:0/871159377 shutdown_connections 2024-04-25T15:22:04.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.263+0000 7f18eb684640 1 -- 172.21.15.146:0/871159377 wait complete. 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.265+0000 7f18eb684640 1 Processor -- start 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.265+0000 7f18eb684640 1 -- start start 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.266+0000 7f18eb684640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d0004800 0x7f18e409f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.266+0000 7f18eb684640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18e41596f0 0x7f18e4159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.266+0000 7f18eb684640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 0x7f18e41627b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.266+0000 7f18eb684640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f18e4070980 con 0x7f18e41623a0 2024-04-25T15:22:04.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.266+0000 7f18eb684640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18e40cf4b0 con 0x7f18e41596f0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.266+0000 7f18eb684640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f18e4070360 con 0x7f18d0004800 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 0x7f18e41627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 0x7f18e41627b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43700/0 (socket says 172.21.15.146:43700) 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 -- 172.21.15.146:0/2593467657 learned_addr learned my addr 172.21.15.146:0/2593467657 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9420640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d0004800 0x7f18e409f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e8c1f640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18e41596f0 0x7f18e4159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 -- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d0004800 msgr2=0x7f18e409f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d0004800 0x7f18e409f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 -- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18e41596f0 msgr2=0x7f18e4159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18e41596f0 0x7f18e4159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 -- 172.21.15.146:0/2593467657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e4162cf0 con 0x7f18e41623a0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18e9c21640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 0x7f18e41627b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f18e40cf8a0 tx=0x7f18dc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc015430 con 0x7f18e41623a0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18dc01d070 con 0x7f18e41623a0 2024-04-25T15:22:04.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc015810 con 0x7f18e41623a0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.267+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e415a3a0 con 0x7f18e41623a0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.268+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e415a960 con 0x7f18e41623a0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.268+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e406a0e0 con 0x7f18e41623a0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.269+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f18dc0159b0 con 0x7f18e41623a0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.269+0000 7f18d9ffb640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f18c8074540 0x7f18c8076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.270+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f18dc095e40 con 0x7f18e41623a0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.270+0000 7f18e9420640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f18c8074540 0x7f18c8076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.270+0000 7f18e9420640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f18c8074540 0x7f18c8076a50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f18e0001050 tx=0x7f18e0000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.274+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18e4162cf0 con 0x7f18e41623a0 2024-04-25T15:22:04.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.489+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18e40711b0 con 0x7f18e41623a0 2024-04-25T15:22:04.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.489+0000 7f18d9ffb640 1 -- 172.21.15.146:0/2593467657 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5192 (secure 0 0 0) 0x7f18dc010040 con 0x7f18e41623a0 2024-04-25T15:22:04.490 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:04.490 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:53.503394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5553,"qdb_cluster":[5553]},"id":2}]} 2024-04-25T15:22:04.491 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-25T15:22:04.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f18c8074540 msgr2=0x7f18c8076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f18c8074540 0x7f18c8076a50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f18e0001050 tx=0x7f18e0000f80 comp rx=0 tx=0).stop 2024-04-25T15:22:04.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 msgr2=0x7f18e41627b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 0x7f18e41627b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f18e40cf8a0 tx=0x7f18dc00f580 comp rx=0 tx=0).stop 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 shutdown_connections 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f18c8074540 0x7f18c8076a50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18e41623a0 0x7f18e41627b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18e41596f0 0x7f18e4159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 --2- 172.21.15.146:0/2593467657 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f18d0004800 0x7f18e409f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 >> 172.21.15.146:0/2593467657 conn(0x7f18e409b2b0 msgr2=0x7f18e415d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 shutdown_connections 2024-04-25T15:22:04.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.491+0000 7f18eb684640 1 -- 172.21.15.146:0/2593467657 wait complete. 2024-04-25T15:22:04.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:04.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.612+0000 7f582d355640 1 Processor -- start 2024-04-25T15:22:04.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.612+0000 7f582d355640 1 -- start start 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f582d355640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58280d4f60 0x7f58280d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f582d355640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58280b9200 con 0x7f58280d0020 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f582d355640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58280d1070 con 0x7f582806ccc0 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f582d355640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f582806c550 con 0x7f58280d4f60 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f5826ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58280d4f60 0x7f58280d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f58267fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f58280d0020 0x7f58280d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52212/0 (socket says 172.21.15.146:52212) 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f58267fc640 1 -- 172.21.15.146:0/1190158459 learned_addr learned my addr 172.21.15.146:0/1190158459 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f5826ffd640 1 -- 172.21.15.146:0/1190158459 >> v1:172.21.15.177:6789/0 conn(0x7f582806ccc0 legacy=0x7f582806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f5826ffd640 1 -- 172.21.15.146:0/1190158459 >> v1:172.21.15.146:6789/0 conn(0x7f58280d0020 legacy=0x7f58280d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f5826ffd640 1 -- 172.21.15.146:0/1190158459 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58280d1310 con 0x7f58280d4f60 2024-04-25T15:22:04.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.614+0000 7f5825ffb640 1 -- 172.21.15.146:0/1190158459 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401446701 0 0) 0x7f581c001110 con 0x7f58280d0020 2024-04-25T15:22:04.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.615+0000 7f5826ffd640 1 --2- 172.21.15.146:0/1190158459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f58280d5370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f58180097a0 tx=0x7f581802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbb68c232c84256 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.615+0000 7f5825ffb640 1 -- 172.21.15.146:0/1190158459 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818037070 con 0x7f58280d4f60 2024-04-25T15:22:04.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.615+0000 7f5825ffb640 1 -- 172.21.15.146:0/1190158459 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5818032440 con 0x7f58280d4f60 2024-04-25T15:22:04.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.615+0000 7f5825ffb640 1 -- 172.21.15.146:0/1190158459 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581802ae60 con 0x7f58280d4f60 2024-04-25T15:22:04.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 -- 172.21.15.146:0/1190158459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 msgr2=0x7f58280d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 --2- 172.21.15.146:0/1190158459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f58280d5370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f58180097a0 tx=0x7f581802a110 comp rx=0 tx=0).stop 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 -- 172.21.15.146:0/1190158459 shutdown_connections 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 --2- 172.21.15.146:0/1190158459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f58280d5370 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 -- 172.21.15.146:0/1190158459 >> 172.21.15.146:0/1190158459 conn(0x7f58280d2450 msgr2=0x7f58280d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 -- 172.21.15.146:0/1190158459 shutdown_connections 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.616+0000 7f582d355640 1 -- 172.21.15.146:0/1190158459 wait complete. 2024-04-25T15:22:04.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.618+0000 7f582d355640 1 Processor -- start 2024-04-25T15:22:04.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.618+0000 7f582d355640 1 -- start start 2024-04-25T15:22:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f582d355640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582806ccc0 0x7f58280c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f582d355640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58280d0020 0x7f582815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f582d355640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f582815cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f582d355640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f58280b5150 con 0x7f58280d4f60 2024-04-25T15:22:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f582d355640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f582806c830 con 0x7f58280d0020 2024-04-25T15:22:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f582d355640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f58280b52d0 con 0x7f582806ccc0 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f582815cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f582815cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43706/0 (socket says 172.21.15.146:43706) 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 -- 172.21.15.146:0/1163419484 learned_addr learned my addr 172.21.15.146:0/1163419484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f5826ffd640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582806ccc0 0x7f58280c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58267fc640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58280d0020 0x7f582815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 -- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582806ccc0 msgr2=0x7f58280c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582806ccc0 0x7f58280c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 -- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58280d0020 msgr2=0x7f582815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58280d0020 0x7f582815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58277fe640 1 -- 172.21.15.146:0/1163419484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5828158010 con 0x7f58280d4f60 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.619+0000 7f58267fc640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58280d0020 0x7f582815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:04.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.620+0000 7f58277fe640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f582815cc30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f582000c680 tx=0x7f582000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.620+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5820003b70 con 0x7f58280d4f60 2024-04-25T15:22:04.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.620+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5820003d10 con 0x7f58280d4f60 2024-04-25T15:22:04.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.620+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58280c8580 con 0x7f58280d4f60 2024-04-25T15:22:04.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.620+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5820005310 con 0x7f58280d4f60 2024-04-25T15:22:04.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.620+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5828157a50 con 0x7f58280d4f60 2024-04-25T15:22:04.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.621+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f582806a0e0 con 0x7f58280d4f60 2024-04-25T15:22:04.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.621+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f58200054b0 con 0x7f58280d4f60 2024-04-25T15:22:04.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.622+0000 7f580f7fe640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5814074540 0x7f5814076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.622+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f58200952e0 con 0x7f58280d4f60 2024-04-25T15:22:04.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.622+0000 7f5826ffd640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5814074540 0x7f5814076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.622+0000 7f5826ffd640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5814074540 0x7f5814076a50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5818034000 tx=0x7f581802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.627+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f582005f880 con 0x7f58280d4f60 2024-04-25T15:22:04.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.803+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58280d4100 con 0x7f58280d4f60 2024-04-25T15:22:04.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.804+0000 7f580f7fe640 1 -- 172.21.15.146:0/1163419484 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7f582005f180 con 0x7f58280d4f60 2024-04-25T15:22:04.805 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:04.805 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:04.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5814074540 msgr2=0x7f5814076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5814074540 0x7f5814076a50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5818034000 tx=0x7f581802e000 comp rx=0 tx=0).stop 2024-04-25T15:22:04.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 msgr2=0x7f582815cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f582815cc30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f582000c680 tx=0x7f582000cb40 comp rx=0 tx=0).stop 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 shutdown_connections 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5814074540 0x7f5814076a50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58280d4f60 0x7f582815cc30 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58280d0020 0x7f582815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 --2- 172.21.15.146:0/1163419484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582806ccc0 0x7f58280c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 >> 172.21.15.146:0/1163419484 conn(0x7f58280d2450 msgr2=0x7f58280b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 shutdown_connections 2024-04-25T15:22:04.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.806+0000 7f582d355640 1 -- 172.21.15.146:0/1163419484 wait complete. 2024-04-25T15:22:04.819 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:22:04.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:04.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.926+0000 7f1cb0bba640 1 Processor -- start 2024-04-25T15:22:04.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.926+0000 7f1cb0bba640 1 -- start start 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1cb0bba640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1cac06ee60 0x7f1cac0b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1cb0bba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac0b8eb0 con 0x7f1cac163eb0 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1cb0bba640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac08ce00 con 0x7f1cac1649d0 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1cb0bba640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f1cac161840 con 0x7f1cac06ee60 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca9d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1cac163eb0 0x7f1cac1642c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52226/0 (socket says 172.21.15.146:52226) 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/3495119112 learned_addr learned my addr 172.21.15.146:0/3495119112 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835155768 0 0) 0x7f1cac0b8eb0 con 0x7f1cac163eb0 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1caa59c640 1 --2- 172.21.15.146:0/3495119112 >> v2:172.21.15.177:3300/0 conn(0x7f1cac06ee60 0x7f1cac0b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c94003520 con 0x7f1cac163eb0 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2914268217 0 0) 0x7f1c94003520 con 0x7f1cac163eb0 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 >> v2:172.21.15.177:3300/0 conn(0x7f1cac06ee60 msgr2=0x7f1cac0b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 --2- 172.21.15.146:0/3495119112 >> v2:172.21.15.177:3300/0 conn(0x7f1cac06ee60 0x7f1cac0b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 >> v1:172.21.15.177:6789/0 conn(0x7f1cac1649d0 legacy=0x7f1cac0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac08d0e0 con 0x7f1cac163eb0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f1ca0001670 con 0x7f1cac163eb0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 >> v1:172.21.15.146:6789/0 conn(0x7f1cac163eb0 legacy=0x7f1cac1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c940039b0 0x7f1c94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c94004800 0x7f1c94004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c940051c0 0x7f1c9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.928+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac161840 con 0x7f1c94004800 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9400c160 con 0x7f1c940039b0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9400c360 con 0x7f1c940051c0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c94004800 0x7f1c94004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1caa59c640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c940039b0 0x7f1c94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1caad9d640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c940051c0 0x7f1c9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c940051c0 msgr2=0x7f1c9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:04.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c940051c0 0x7f1c9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c940039b0 msgr2=0x7f1c94003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c940039b0 0x7f1c94003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/3495119112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c94003520 con 0x7f1c94004800 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1caad9d640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c940051c0 0x7f1c9400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.929+0000 7f1caad9d640 1 -- 172.21.15.146:0/3495119112 reap_dead start 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c94004800 0x7f1c94004c10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c240 tx=0x7f1c94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb68613d160c907 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0007d10 con 0x7f1c94004800 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca0005ad0 con 0x7f1c94004800 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1ca959a640 1 -- 172.21.15.146:0/3495119112 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0007070 con 0x7f1c94004800 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1cb0bba640 1 -- 172.21.15.146:0/3495119112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c94004800 msgr2=0x7f1c94004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c94004800 0x7f1c94004c10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c240 tx=0x7f1c94005830 comp rx=0 tx=0).stop 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1cb0bba640 1 -- 172.21.15.146:0/3495119112 shutdown_connections 2024-04-25T15:22:04.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/3495119112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c94004800 0x7f1c94004c10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1cb0bba640 1 -- 172.21.15.146:0/3495119112 >> 172.21.15.146:0/3495119112 conn(0x7f1cac0b52e0 msgr2=0x7f1cac0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:04.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.930+0000 7f1cb0bba640 1 -- 172.21.15.146:0/3495119112 shutdown_connections 2024-04-25T15:22:04.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.931+0000 7f1cb0bba640 1 -- 172.21.15.146:0/3495119112 wait complete. 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.932+0000 7f1cb0bba640 1 Processor -- start 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.932+0000 7f1cb0bba640 1 -- start start 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c94004800 0x7f1cac1537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 0x7f1cac0b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1cac0b7580 0x7f1cac1f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac08d730 con 0x7f1cac0b6c30 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac06f440 con 0x7f1cac0b7580 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1cac08dbc0 con 0x7f1c94004800 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 0x7f1cac0b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 0x7f1cac0b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43734/0 (socket says 172.21.15.146:43734) 2024-04-25T15:22:04.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/904348060 learned_addr learned my addr 172.21.15.146:0/904348060 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1caa59c640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c94004800 0x7f1cac1537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1caad9d640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1cac0b7580 0x7f1cac1f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c94004800 msgr2=0x7f1cac1537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c94004800 0x7f1cac1537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1cac0b7580 msgr2=0x7f1cac1f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1cac0b7580 0x7f1cac1f9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 -- 172.21.15.146:0/904348060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac153cf0 con 0x7f1cac0b6c30 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1caad9d640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1cac0b7580 0x7f1cac1f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1ca9d9b640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 0x7f1cac0b7040 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c240 tx=0x7f1cac08cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0041070 con 0x7f1cac0b6c30 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca0000ed0 con 0x7f1cac0b6c30 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0007d10 con 0x7f1cac0b6c30 2024-04-25T15:22:04.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cac162bb0 con 0x7f1cac0b6c30 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.934+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cac163170 con 0x7f1cac0b6c30 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.935+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cac0b2ef0 con 0x7f1cac0b6c30 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.936+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f1ca0044070 con 0x7f1cac0b6c30 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.937+0000 7f1c9affd640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c8c074570 0x7f1c8c076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.937+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f1ca00b57c0 con 0x7f1cac0b6c30 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.937+0000 7f1caa59c640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c8c074570 0x7f1c8c076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.937+0000 7f1caa59c640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c8c074570 0x7f1c8c076a80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c007520 tx=0x7f1c9c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:04.941+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ca007fd60 con 0x7f1cac0b6c30 2024-04-25T15:22:05.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.152+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cac06a0e0 con 0x7f1cac0b6c30 2024-04-25T15:22:05.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.153+0000 7f1c9affd640 1 -- 172.21.15.146:0/904348060 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5192 (secure 0 0 0) 0x7f1ca007f660 con 0x7f1cac0b6c30 2024-04-25T15:22:05.153 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:05.154 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:53.503394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5553,"qdb_cluster":[5553]},"id":2}]} 2024-04-25T15:22:05.154 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-25T15:22:05.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c8c074570 msgr2=0x7f1c8c076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c8c074570 0x7f1c8c076a80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c007520 tx=0x7f1c9c00b040 comp rx=0 tx=0).stop 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 msgr2=0x7f1cac0b7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 0x7f1cac0b7040 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c240 tx=0x7f1cac08cfd0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 shutdown_connections 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1c8c074570 0x7f1c8c076a80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1cac0b7580 0x7f1cac1f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cac0b6c30 0x7f1cac0b7040 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 --2- 172.21.15.146:0/904348060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c94004800 0x7f1cac1537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 >> 172.21.15.146:0/904348060 conn(0x7f1cac0b52e0 msgr2=0x7f1cac0ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 shutdown_connections 2024-04-25T15:22:05.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.155+0000 7f1cb0bba640 1 -- 172.21.15.146:0/904348060 wait complete. 2024-04-25T15:22:05.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.274+0000 7f96a2013640 1 Processor -- start 2024-04-25T15:22:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.274+0000 7f96a2013640 1 -- start start 2024-04-25T15:22:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f96a2013640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f969c0d4f60 0x7f969c0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f96a2013640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f969c0b9200 con 0x7f969c06ccc0 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f96a2013640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f969c0d1070 con 0x7f969c0d0020 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f96a2013640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f969c06c550 con 0x7f969c0d4f60 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f969bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f969c06ccc0 0x7f969c06d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52242/0 (socket says 172.21.15.146:52242) 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f969bfff640 1 -- 172.21.15.146:0/3636802796 learned_addr learned my addr 172.21.15.146:0/3636802796 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.275+0000 7f969b7fe640 1 --2- 172.21.15.146:0/3636802796 >> v2:172.21.15.177:3300/0 conn(0x7f969c0d4f60 0x7f969c0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279326222 0 0) 0x7f9694001490 con 0x7f969c06ccc0 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9684003520 con 0x7f969c06ccc0 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426681728 0 0) 0x7f969c0d1070 con 0x7f969c0d0020 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9694001490 con 0x7f969c0d0020 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969b7fe640 1 -- 172.21.15.146:0/3636802796 >> v1:172.21.15.177:6789/0 conn(0x7f969c0d0020 legacy=0x7f969c0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969b7fe640 1 -- 172.21.15.146:0/3636802796 >> v1:172.21.15.146:6789/0 conn(0x7f969c06ccc0 legacy=0x7f969c06d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.276+0000 7f969b7fe640 1 -- 172.21.15.146:0/3636802796 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969c0d1310 con 0x7f969c0d4f60 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f969b7fe640 1 --2- 172.21.15.146:0/3636802796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c0d4f60 0x7f969c0d5370 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f968c0097a0 tx=0x7f968c02a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82de0d3e74f92159 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c037070 con 0x7f969c0d4f60 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f968c032440 con 0x7f969c0d4f60 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f969a7fc640 1 -- 172.21.15.146:0/3636802796 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c02ae60 con 0x7f969c0d4f60 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f96a2013640 1 -- 172.21.15.146:0/3636802796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c0d4f60 msgr2=0x7f969c0d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f96a2013640 1 --2- 172.21.15.146:0/3636802796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c0d4f60 0x7f969c0d5370 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f968c0097a0 tx=0x7f968c02a110 comp rx=0 tx=0).stop 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f96a2013640 1 -- 172.21.15.146:0/3636802796 shutdown_connections 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f96a2013640 1 --2- 172.21.15.146:0/3636802796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c0d4f60 0x7f969c0d5370 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f96a2013640 1 -- 172.21.15.146:0/3636802796 >> 172.21.15.146:0/3636802796 conn(0x7f969c0d2450 msgr2=0x7f969c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.277+0000 7f96a2013640 1 -- 172.21.15.146:0/3636802796 shutdown_connections 2024-04-25T15:22:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.278+0000 7f96a2013640 1 -- 172.21.15.146:0/3636802796 wait complete. 2024-04-25T15:22:05.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.279+0000 7f96a2013640 1 Processor -- start 2024-04-25T15:22:05.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.280+0000 7f96a2013640 1 -- start start 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f96a2013640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c06ccc0 0x7f969c0cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f96a2013640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 0x7f969c14bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f96a2013640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f969c0d4f60 0x7f969c162cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f96a2013640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f969c0b5150 con 0x7f969c0d0020 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f96a2013640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f969c06c830 con 0x7f969c06ccc0 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f96a2013640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f969c0b52d0 con 0x7f969c0d4f60 2024-04-25T15:22:05.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 0x7f969c14bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 0x7f969c14bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43742/0 (socket says 172.21.15.146:43742) 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 -- 172.21.15.146:0/1950778803 learned_addr learned my addr 172.21.15.146:0/1950778803 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969bfff640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f969c0d4f60 0x7f969c162cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969b7fe640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c06ccc0 0x7f969c0cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 -- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f969c0d4f60 msgr2=0x7f969c162cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f969c0d4f60 0x7f969c162cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 -- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c06ccc0 msgr2=0x7f969c0cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c06ccc0 0x7f969c0cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 -- 172.21.15.146:0/1950778803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969c14f370 con 0x7f969c0d0020 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.281+0000 7f969affd640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 0x7f969c14bc10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f969000c430 tx=0x7f969000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.282+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690017070 con 0x7f969c0d0020 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.282+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96900052f0 con 0x7f969c0d0020 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.282+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690003830 con 0x7f969c0d0020 2024-04-25T15:22:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.282+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969c14ead0 con 0x7f969c0d0020 2024-04-25T15:22:05.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.282+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f969c14f090 con 0x7f969c0d0020 2024-04-25T15:22:05.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.283+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f969c0769e0 con 0x7f969c0d0020 2024-04-25T15:22:05.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.283+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f969001a070 con 0x7f969c0d0020 2024-04-25T15:22:05.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.284+0000 7f9683fff640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9688074570 0x7f9688076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.284+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f9690094700 con 0x7f969c0d0020 2024-04-25T15:22:05.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.284+0000 7f969b7fe640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9688074570 0x7f9688076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.284+0000 7f969b7fe640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9688074570 0x7f9688076a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f968c034000 tx=0x7f968c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.288+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f969c0769e0 con 0x7f969c0d0020 2024-04-25T15:22:05.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.501+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f969c06a0e0 con 0x7f969c0d0020 2024-04-25T15:22:05.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.502+0000 7f9683fff640 1 -- 172.21.15.146:0/1950778803 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5192 (secure 0 0 0) 0x7f969005e650 con 0x7f969c0d0020 2024-04-25T15:22:05.502 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:05.503 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:21:07.618988+0000","modified":"2024-04-25T15:21:53.503394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5553":{"gid":5553,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.177:6834/377672985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":377672985},{"type":"v1","addr":"172.21.15.177:6834","nonce":377672985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5553,"qdb_cluster":[5553]},"id":2}]} 2024-04-25T15:22:05.503 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-25T15:22:05.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.503+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9688074570 msgr2=0x7f9688076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.503+0000 7f96a2013640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9688074570 0x7f9688076a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f968c034000 tx=0x7f968c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 msgr2=0x7f969c14bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 0x7f969c14bc10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f969000c430 tx=0x7f969000c8f0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 shutdown_connections 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9688074570 0x7f9688076a80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f969c0d4f60 0x7f969c162cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f969c0d0020 0x7f969c14bc10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 --2- 172.21.15.146:0/1950778803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969c06ccc0 0x7f969c0cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 >> 172.21.15.146:0/1950778803 conn(0x7f969c0d2450 msgr2=0x7f969c06d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 shutdown_connections 2024-04-25T15:22:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.504+0000 7f96a2013640 1 -- 172.21.15.146:0/1950778803 wait complete. 2024-04-25T15:22:05.516 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:05.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.623+0000 7ffa1a2bd640 1 Processor -- start 2024-04-25T15:22:05.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.623+0000 7ffa1a2bd640 1 -- start start 2024-04-25T15:22:05.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1a2bd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffa140d4f60 0x7ffa140d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1a2bd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffa140d0020 0x7ffa140d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1a2bd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa140b9200 con 0x7ffa14163d80 2024-04-25T15:22:05.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1a2bd640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffa1406c550 con 0x7ffa140d0020 2024-04-25T15:22:05.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1a2bd640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ffa1406c830 con 0x7ffa140d4f60 2024-04-25T15:22:05.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa137fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffa140d0020 0x7ffa140d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1885a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffa14163d80 0x7ffa14164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52250/0 (socket says 172.21.15.146:52250) 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.624+0000 7ffa1885a640 1 -- 172.21.15.146:0/618056596 learned_addr learned my addr 172.21.15.146:0/618056596 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa13fff640 1 --2- 172.21.15.146:0/618056596 >> v2:172.21.15.177:3300/0 conn(0x7ffa140d4f60 0x7ffa140d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa137fe640 1 -- 172.21.15.146:0/618056596 >> v2:172.21.15.177:3300/0 conn(0x7ffa140d4f60 msgr2=0x7ffa140d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa137fe640 1 --2- 172.21.15.146:0/618056596 >> v2:172.21.15.177:3300/0 conn(0x7ffa140d4f60 0x7ffa140d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa137fe640 1 -- 172.21.15.146:0/618056596 >> v1:172.21.15.146:6789/0 conn(0x7ffa14163d80 legacy=0x7ffa14164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa137fe640 1 -- 172.21.15.146:0/618056596 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa140d0970 con 0x7ffa140d0020 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa12ffd640 1 -- 172.21.15.146:0/618056596 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70927730 0 0) 0x7ffa0c001490 con 0x7ffa14163d80 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.625+0000 7ffa137fe640 1 --2- 172.21.15.146:0/618056596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d0020 0x7ffa140d0430 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffa140d5930 tx=0x7ffa0802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cedf5dd61b6a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa12ffd640 1 -- 172.21.15.146:0/618056596 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08037070 con 0x7ffa140d0020 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa12ffd640 1 -- 172.21.15.146:0/618056596 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa0802add0 con 0x7ffa140d0020 2024-04-25T15:22:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa12ffd640 1 -- 172.21.15.146:0/618056596 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08032660 con 0x7ffa140d0020 2024-04-25T15:22:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/618056596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d0020 msgr2=0x7ffa140d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/618056596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d0020 0x7ffa140d0430 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffa140d5930 tx=0x7ffa0802a080 comp rx=0 tx=0).stop 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/618056596 shutdown_connections 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/618056596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d0020 0x7ffa140d0430 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/618056596 >> v2:172.21.15.177:3300/0 conn(0x7ffa140d4f60 0x7ffa140d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/618056596 >> 172.21.15.146:0/618056596 conn(0x7ffa140d2450 msgr2=0x7ffa140d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/618056596 shutdown_connections 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.626+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/618056596 wait complete. 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.628+0000 7ffa1a2bd640 1 Processor -- start 2024-04-25T15:22:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.628+0000 7ffa1a2bd640 1 -- start start 2024-04-25T15:22:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.629+0000 7ffa1a2bd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa140d0020 0x7ffa14156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.629+0000 7ffa1a2bd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 0x7ffa141fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.629+0000 7ffa1a2bd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa14163d80 0x7ffa1406d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.629+0000 7ffa1a2bd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffa140d15f0 con 0x7ffa140d4f60 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.629+0000 7ffa1a2bd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa140d1070 con 0x7ffa140d0020 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.629+0000 7ffa1a2bd640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffa140b5150 con 0x7ffa14163d80 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 0x7ffa141fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 0x7ffa141fa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43766/0 (socket says 172.21.15.146:43766) 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 -- 172.21.15.146:0/190115698 learned_addr learned my addr 172.21.15.146:0/190115698 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa1885a640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa14163d80 0x7ffa1406d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa13fff640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa140d0020 0x7ffa14156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 -- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa14163d80 msgr2=0x7ffa1406d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa14163d80 0x7ffa1406d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 -- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa140d0020 msgr2=0x7ffa14156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa140d0020 0x7ffa14156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 -- 172.21.15.146:0/190115698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa14162450 con 0x7ffa140d4f60 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa137fe640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 0x7ffa141fa5c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ffa08035040 tx=0x7ffa080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08037050 con 0x7ffa140d4f60 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa08038040 con 0x7ffa140d4f60 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08004270 con 0x7ffa140d4f60 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa14162730 con 0x7ffa140d4f60 2024-04-25T15:22:05.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.630+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa14162cf0 con 0x7ffa140d4f60 2024-04-25T15:22:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.631+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa14163340 con 0x7ffa140d4f60 2024-04-25T15:22:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.632+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7ffa08003610 con 0x7ffa140d4f60 2024-04-25T15:22:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.632+0000 7ffa10ff9640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa00074540 0x7ffa00076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.632+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7ffa080b5a40 con 0x7ffa140d4f60 2024-04-25T15:22:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.632+0000 7ffa13fff640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa00074540 0x7ffa00076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.633+0000 7ffa13fff640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa00074540 0x7ffa00076a50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa140b4880 tx=0x7ffa04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.637+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa140d0970 con 0x7ffa140d4f60 2024-04-25T15:22:05.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.814+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa141621b0 con 0x7ffa140d4f60 2024-04-25T15:22:05.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.815+0000 7ffa10ff9640 1 -- 172.21.15.146:0/190115698 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17112 (secure 0 0 0) 0x7ffa141621b0 con 0x7ffa140d4f60 2024-04-25T15:22:05.816 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:05.816 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":42,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:21:49.486232+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:21:06.188796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T15:21:06.574527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:05.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa00074540 msgr2=0x7ffa00076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa00074540 0x7ffa00076a50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa140b4880 tx=0x7ffa04008040 comp rx=0 tx=0).stop 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 msgr2=0x7ffa141fa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 0x7ffa141fa5c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ffa08035040 tx=0x7ffa080046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 shutdown_connections 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffa00074540 0x7ffa00076a50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa14163d80 0x7ffa1406d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa140d4f60 0x7ffa141fa5c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 --2- 172.21.15.146:0/190115698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa140d0020 0x7ffa14156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.817+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 >> 172.21.15.146:0/190115698 conn(0x7ffa140d2450 msgr2=0x7ffa140b93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.818+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 shutdown_connections 2024-04-25T15:22:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.818+0000 7ffa1a2bd640 1 -- 172.21.15.146:0/190115698 wait complete. 2024-04-25T15:22:05.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:22:05.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.938+0000 7f62a7fee640 1 Processor -- start 2024-04-25T15:22:05.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.938+0000 7f62a7fee640 1 -- start start 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a7fee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f62a00b7350 0x7f62a00715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a7fee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0160000 0x7f62a006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a7fee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62a0070190 0x7f62a00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a7fee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f62a00705a0 con 0x7f62a0070190 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a7fee640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f62a0071bc0 con 0x7f62a00b7350 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a7fee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f62a00d1b00 con 0x7f62a0160000 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62a0070190 0x7f62a00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62a0070190 0x7f62a00ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43778/0 (socket says 172.21.15.146:43778) 2024-04-25T15:22:05.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 -- 172.21.15.146:0/1057692468 learned_addr learned my addr 172.21.15.146:0/1057692468 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a5d8a640 1 --2- 172.21.15.146:0/1057692468 >> v2:172.21.15.177:3300/0 conn(0x7f62a00b7350 0x7f62a00715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a5589640 1 --2- 172.21.15.146:0/1057692468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0160000 0x7f62a006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 -- 172.21.15.146:0/1057692468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0160000 msgr2=0x7f62a006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 --2- 172.21.15.146:0/1057692468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0160000 0x7f62a006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 -- 172.21.15.146:0/1057692468 >> v2:172.21.15.177:3300/0 conn(0x7f62a00b7350 msgr2=0x7f62a00715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 --2- 172.21.15.146:0/1057692468 >> v2:172.21.15.177:3300/0 conn(0x7f62a00b7350 0x7f62a00715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a658b640 1 -- 172.21.15.146:0/1057692468 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a00d1de0 con 0x7f62a0070190 2024-04-25T15:22:05.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.940+0000 7f62a5d8a640 1 --2- 172.21.15.146:0/1057692468 >> v2:172.21.15.177:3300/0 conn(0x7f62a00b7350 0x7f62a00715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:05.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.941+0000 7f62a658b640 1 --2- 172.21.15.146:0/1057692468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a0070190 0x7f62a00ce4a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f62a0074600 tx=0x7f629802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acebfc9abe9428c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.941+0000 7f62a4d88640 1 -- 172.21.15.146:0/1057692468 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298036070 con 0x7f62a0070190 2024-04-25T15:22:05.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.941+0000 7f62a4d88640 1 -- 172.21.15.146:0/1057692468 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6298031600 con 0x7f62a0070190 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.941+0000 7f62a4d88640 1 -- 172.21.15.146:0/1057692468 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62980319d0 con 0x7f62a0070190 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 -- 172.21.15.146:0/1057692468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a0070190 msgr2=0x7f62a00ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 --2- 172.21.15.146:0/1057692468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a0070190 0x7f62a00ce4a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f62a0074600 tx=0x7f629802a610 comp rx=0 tx=0).stop 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 -- 172.21.15.146:0/1057692468 shutdown_connections 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 --2- 172.21.15.146:0/1057692468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a0070190 0x7f62a00ce4a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 --2- 172.21.15.146:0/1057692468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0160000 0x7f62a006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 --2- 172.21.15.146:0/1057692468 >> v2:172.21.15.177:3300/0 conn(0x7f62a00b7350 0x7f62a00715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 -- 172.21.15.146:0/1057692468 >> 172.21.15.146:0/1057692468 conn(0x7f62a009b2b0 msgr2=0x7f62a00b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 -- 172.21.15.146:0/1057692468 shutdown_connections 2024-04-25T15:22:05.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.942+0000 7f62a7fee640 1 -- 172.21.15.146:0/1057692468 wait complete. 2024-04-25T15:22:05.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.944+0000 7f62a7fee640 1 Processor -- start 2024-04-25T15:22:05.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.944+0000 7f62a7fee640 1 -- start start 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a7fee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0070190 0x7f62a01e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a7fee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 0x7f62a014e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a7fee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62a0160000 0x7f62a014eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a7fee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f62a0070980 con 0x7f62a00b7350 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a7fee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62a00d23a0 con 0x7f62a0160000 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a7fee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f62a0160410 con 0x7f62a0070190 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 0x7f62a014e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 0x7f62a014e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43782/0 (socket says 172.21.15.146:43782) 2024-04-25T15:22:05.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 -- 172.21.15.146:0/3778640266 learned_addr learned my addr 172.21.15.146:0/3778640266 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5d8a640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0070190 0x7f62a01e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a658b640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62a0160000 0x7f62a014eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 -- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0070190 msgr2=0x7f62a01e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0070190 0x7f62a01e72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 -- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62a0160000 msgr2=0x7f62a014eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62a0160000 0x7f62a014eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a5589640 1 -- 172.21.15.146:0/3778640266 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a01e7ba0 con 0x7f62a00b7350 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.945+0000 7f62a658b640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62a0160000 0x7f62a014eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.946+0000 7f62a5589640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 0x7f62a014e3a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f629000d650 tx=0x7f629000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.946+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629000b6c0 con 0x7f62a00b7350 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.946+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290019070 con 0x7f62a00b7350 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.946+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a01e7e80 con 0x7f62a00b7350 2024-04-25T15:22:05.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.946+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629000baa0 con 0x7f62a00b7350 2024-04-25T15:22:05.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.946+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a01e8440 con 0x7f62a00b7350 2024-04-25T15:22:05.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.947+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a006a0e0 con 0x7f62a00b7350 2024-04-25T15:22:05.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.947+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f629000bc40 con 0x7f62a00b7350 2024-04-25T15:22:05.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.948+0000 7f62967fc640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6284074540 0x7f6284076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:05.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.948+0000 7f62a5d8a640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6284074540 0x7f6284076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:05.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.948+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f6290095810 con 0x7f62a00b7350 2024-04-25T15:22:05.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.948+0000 7f62a5d8a640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6284074540 0x7f6284076a50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f629c006fd0 tx=0x7f629c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:05.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:05.952+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f629005fe40 con 0x7f62a00b7350 2024-04-25T15:22:06.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.164+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f62a00711b0 con 0x7f62a00b7350 2024-04-25T15:22:06.165 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-04-25T15:22:06.164+0000 7f79638be640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:22:06.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.166+0000 7f62967fc640 1 -- 172.21.15.146:0/3778640266 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7f629005f740 con 0x7f62a00b7350 2024-04-25T15:22:06.167 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:22:06.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.169+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6284074540 msgr2=0x7f6284076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.169+0000 7f62a7fee640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6284074540 0x7f6284076a50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f629c006fd0 tx=0x7f629c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.169+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 msgr2=0x7f62a014e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.169+0000 7f62a7fee640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 0x7f62a014e3a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f629000d650 tx=0x7f629000db10 comp rx=0 tx=0).stop 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 shutdown_connections 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6284074540 0x7f6284076a50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62a0160000 0x7f62a014eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62a00b7350 0x7f62a014e3a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 --2- 172.21.15.146:0/3778640266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f62a0070190 0x7f62a01e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 >> 172.21.15.146:0/3778640266 conn(0x7f62a009b2b0 msgr2=0x7f62a006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 shutdown_connections 2024-04-25T15:22:06.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.170+0000 7f62a7fee640 1 -- 172.21.15.146:0/3778640266 wait complete. 2024-04-25T15:22:06.176 INFO:tasks.ceph.mds.d.smithi177.stderr: -487> 2024-04-25T15:21:58.247+0000 7ff855cb1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:22:06.182 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:22:06.228 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-04-25T15:22:06.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.293+0000 7f9f5257a640 1 Processor -- start 2024-04-25T15:22:06.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.293+0000 7f9f5257a640 1 -- start start 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f5257a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0ccd40 0x7f9f4c09bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f5257a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f4c0d2680 0x7f9f4c0d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f5257a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f4c0d2fd0 con 0x7f9f4c0b2400 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f5257a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9f4c0b1960 con 0x7f9f4c0d2680 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f5257a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9f4c0b3ce0 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f4c0d2680 0x7f9f4c0d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f4c0d2680 0x7f9f4c0d2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43788/0 (socket says 172.21.15.146:43788) 2024-04-25T15:22:06.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/2073476609 learned_addr learned my addr 172.21.15.146:0/2073476609 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:06.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4bfff640 1 --2- 172.21.15.146:0/2073476609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0ccd40 0x7f9f4c09bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:06.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/2073476609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0ccd40 msgr2=0x7f9f4c09bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:06.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4b7fe640 1 --2- 172.21.15.146:0/2073476609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0ccd40 0x7f9f4c09bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.294+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/2073476609 >> v1:172.21.15.177:6789/0 conn(0x7f9f4c0b2400 legacy=0x7f9f4c0b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:06.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.295+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/2073476609 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c0b1c00 con 0x7f9f4c0d2680 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.295+0000 7f9f4b7fe640 1 --2- 172.21.15.146:0/2073476609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c0d2a90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009870 tx=0x7f9f4002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899ac9f47c18d6de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.295+0000 7f9f4affd640 1 -- 172.21.15.146:0/2073476609 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40037070 con 0x7f9f4c0d2680 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.295+0000 7f9f4affd640 1 -- 172.21.15.146:0/2073476609 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f40032440 con 0x7f9f4c0d2680 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.295+0000 7f9f4affd640 1 -- 172.21.15.146:0/2073476609 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40032810 con 0x7f9f4c0d2680 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 -- 172.21.15.146:0/2073476609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0d2680 msgr2=0x7f9f4c0d2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 --2- 172.21.15.146:0/2073476609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c0d2a90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009870 tx=0x7f9f4002a270 comp rx=0 tx=0).stop 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 -- 172.21.15.146:0/2073476609 shutdown_connections 2024-04-25T15:22:06.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 --2- 172.21.15.146:0/2073476609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c0d2a90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 --2- 172.21.15.146:0/2073476609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0ccd40 0x7f9f4c09bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 -- 172.21.15.146:0/2073476609 >> 172.21.15.146:0/2073476609 conn(0x7f9f4c0b7400 msgr2=0x7f9f4c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:06.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 -- 172.21.15.146:0/2073476609 shutdown_connections 2024-04-25T15:22:06.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.296+0000 7f9f5257a640 1 -- 172.21.15.146:0/2073476609 wait complete. 2024-04-25T15:22:06.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.298+0000 7f9f5257a640 1 Processor -- start 2024-04-25T15:22:06.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.298+0000 7f9f5257a640 1 -- start start 2024-04-25T15:22:06.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f5257a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0b2400 0x7f9f4c159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:06.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f5257a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 0x7f9f4c154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:06.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f5257a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:06.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f5257a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c0b3940 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f5257a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c0b3fc0 con 0x7f9f4c0d2680 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f5257a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9f4c0b3ac0 con 0x7f9f4c0b2400 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 0x7f9f4c154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 0x7f9f4c154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43800/0 (socket says 172.21.15.146:43800) 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.299+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/4022731009 learned_addr learned my addr 172.21.15.146:0/4022731009 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4bfff640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0b2400 0x7f9f4c159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f50b17640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0b2400 msgr2=0x7f9f4c159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4b7fe640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0b2400 0x7f9f4c159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f4c0d2680 msgr2=0x7f9f4c160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4b7fe640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4b7fe640 1 -- 172.21.15.146:0/4022731009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c162240 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f4b7fe640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 0x7f9f4c154dd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9f40000c00 tx=0x7f9f400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:06.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40037040 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f40042070 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4c162520 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40030880 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.300+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f4c15d710 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.301+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f4c0cffa0 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.302+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f9f40030a20 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.302+0000 7f9f48ff9640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f38074570 0x7f9f38076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:06.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.302+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f9f400b5cd0 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.302+0000 7f9f4bfff640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f38074570 0x7f9f38076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:06.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.303+0000 7f9f4bfff640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f38074570 0x7f9f38076a80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0b8040 tx=0x7f9f3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:06.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.307+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f4c0cffa0 con 0x7f9f4c0ccd40 2024-04-25T15:22:06.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:06.517+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9f4c06ef10 con 0x7f9f4c0ccd40 2024-04-25T15:22:07.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.168+0000 7f9f48ff9640 1 -- 172.21.15.146:0/4022731009 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f9f4c06ef10 con 0x7f9f4c0ccd40 2024-04-25T15:22:07.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f38074570 msgr2=0x7f9f38076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:07.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f38074570 0x7f9f38076a80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0b8040 tx=0x7f9f3c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:07.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 msgr2=0x7f9f4c154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:07.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 0x7f9f4c154dd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9f40000c00 tx=0x7f9f400046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 shutdown_connections 2024-04-25T15:22:07.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f38074570 0x7f9f38076a80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f4c0d2680 0x7f9f4c160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f4c0ccd40 0x7f9f4c154dd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 --2- 172.21.15.146:0/4022731009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f4c0b2400 0x7f9f4c159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 >> 172.21.15.146:0/4022731009 conn(0x7f9f4c0b7400 msgr2=0x7f9f4c0d1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:07.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 shutdown_connections 2024-04-25T15:22:07.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.170+0000 7f9f5257a640 1 -- 172.21.15.146:0/4022731009 wait complete. 2024-04-25T15:22:07.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:22:07.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.300+0000 7f6682382640 1 Processor -- start 2024-04-25T15:22:07.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.300+0000 7f6682382640 1 -- start start 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f6682382640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f667c0b3060 0x7f667c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f6682382640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f667c0d1e00 0x7f667c0d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f6682382640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c0d2750 con 0x7f667c15de20 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f6682382640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f667c06c880 con 0x7f667c0b3060 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f6682382640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f667c06cb60 con 0x7f667c0d1e00 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f667bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f667c0b3060 0x7f667c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:07.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f667bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f667c0b3060 0x7f667c0b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43804/0 (socket says 172.21.15.146:43804) 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.301+0000 7f667bfff640 1 -- 172.21.15.146:0/2867965940 learned_addr learned my addr 172.21.15.146:0/2867965940 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.302+0000 7f667b7fe640 1 --2- 172.21.15.146:0/2867965940 >> v2:172.21.15.177:3300/0 conn(0x7f667c0d1e00 0x7f667c0d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.302+0000 7f667bfff640 1 -- 172.21.15.146:0/2867965940 >> v2:172.21.15.177:3300/0 conn(0x7f667c0d1e00 msgr2=0x7f667c0d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.302+0000 7f667bfff640 1 --2- 172.21.15.146:0/2867965940 >> v2:172.21.15.177:3300/0 conn(0x7f667c0d1e00 0x7f667c0d2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.302+0000 7f667bfff640 1 -- 172.21.15.146:0/2867965940 >> v1:172.21.15.177:6789/0 conn(0x7f667c15de20 legacy=0x7f667c15e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.302+0000 7f667bfff640 1 -- 172.21.15.146:0/2867965940 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c06ce40 con 0x7f667c0b3060 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.302+0000 7f667bfff640 1 --2- 172.21.15.146:0/2867965940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c0b3470 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f667c0cc6a0 tx=0x7f666c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa2b78428f3a4f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f667affd640 1 -- 172.21.15.146:0/2867965940 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c037070 con 0x7f667c0b3060 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f667affd640 1 -- 172.21.15.146:0/2867965940 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c02ae50 con 0x7f667c0b3060 2024-04-25T15:22:07.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f667affd640 1 -- 172.21.15.146:0/2867965940 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c032860 con 0x7f667c0b3060 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 -- 172.21.15.146:0/2867965940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 msgr2=0x7f667c0b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 --2- 172.21.15.146:0/2867965940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c0b3470 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f667c0cc6a0 tx=0x7f666c02a300 comp rx=0 tx=0).stop 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 -- 172.21.15.146:0/2867965940 shutdown_connections 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 --2- 172.21.15.146:0/2867965940 >> v2:172.21.15.177:3300/0 conn(0x7f667c0d1e00 0x7f667c0d2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 --2- 172.21.15.146:0/2867965940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c0b3470 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 -- 172.21.15.146:0/2867965940 >> 172.21.15.146:0/2867965940 conn(0x7f667c09b2b0 msgr2=0x7f667c0b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 -- 172.21.15.146:0/2867965940 shutdown_connections 2024-04-25T15:22:07.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.303+0000 7f6682382640 1 -- 172.21.15.146:0/2867965940 wait complete. 2024-04-25T15:22:07.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.305+0000 7f6682382640 1 Processor -- start 2024-04-25T15:22:07.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.305+0000 7f6682382640 1 -- start start 2024-04-25T15:22:07.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:07.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f667c0d1e00 0x7f667c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:07.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f667c15de20 0x7f667c1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:07.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f667c06c550 con 0x7f667c0b3060 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f667c06d400 con 0x7f667c15de20 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f667c06d920 con 0x7f667c0d1e00 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43808/0 (socket says 172.21.15.146:43808) 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 -- 172.21.15.146:0/3874063726 learned_addr learned my addr 172.21.15.146:0/3874063726 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 -- 172.21.15.146:0/3874063726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f667c0d1e00 msgr2=0x7f667c157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f667c0d1e00 0x7f667c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 -- 172.21.15.146:0/3874063726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f667c15de20 msgr2=0x7f667c1554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f667c15de20 0x7f667c1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 -- 172.21.15.146:0/3874063726 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c152df0 con 0x7f667c0b3060 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f667bfff640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c150370 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f666c000c00 tx=0x7f666c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c041020 con 0x7f667c0b3060 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c03c670 con 0x7f667c0b3060 2024-04-25T15:22:07.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667c1530d0 con 0x7f667c0b3060 2024-04-25T15:22:07.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c037040 con 0x7f667c0b3060 2024-04-25T15:22:07.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.307+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667c153690 con 0x7f667c0b3060 2024-04-25T15:22:07.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.308+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667c06a0e0 con 0x7f667c0b3060 2024-04-25T15:22:07.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.309+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f666c044020 con 0x7f667c0b3060 2024-04-25T15:22:07.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.310+0000 7f6678ff9640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6668074570 0x7f6668076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:07.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.310+0000 7f667b7fe640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6668074570 0x7f6668076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:07.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.310+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f666c0b5a70 con 0x7f667c0b3060 2024-04-25T15:22:07.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.310+0000 7f667b7fe640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6668074570 0x7f6668076a80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6670006fd0 tx=0x7f6670008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:07.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.315+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f666c04b070 con 0x7f667c0b3060 2024-04-25T15:22:07.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:07.490+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f667c070ae0 con 0x7f667c0b3060 2024-04-25T15:22:08.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.169+0000 7f6678ff9640 1 -- 172.21.15.146:0/3874063726 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) ==== 181+0+0 (secure 0 0 0) 0x7f666c07ffc0 con 0x7f667c0b3060 2024-04-25T15:22:08.170 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:22:08.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6668074570 msgr2=0x7f6668076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6668074570 0x7f6668076a80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6670006fd0 tx=0x7f6670008040 comp rx=0 tx=0).stop 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 msgr2=0x7f667c150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c150370 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f666c000c00 tx=0x7f666c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 shutdown_connections 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6668074570 0x7f6668076a80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f667c15de20 0x7f667c1554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f667c0d1e00 0x7f667c157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 --2- 172.21.15.146:0/3874063726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f667c0b3060 0x7f667c150370 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.173+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 >> 172.21.15.146:0/3874063726 conn(0x7f667c09b2b0 msgr2=0x7f667c06fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:08.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.174+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 shutdown_connections 2024-04-25T15:22:08.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.174+0000 7f6682382640 1 -- 172.21.15.146:0/3874063726 wait complete. 2024-04-25T15:22:08.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:22:08.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.303+0000 7f7a29d84640 1 Processor -- start 2024-04-25T15:22:08.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.303+0000 7f7a29d84640 1 -- start start 2024-04-25T15:22:08.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a29d84640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a240d1d60 0x7f7a240d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a29d84640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 0x7f7a24162470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a29d84640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a240b8cc0 con 0x7f7a241629b0 2024-04-25T15:22:08.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a29d84640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2409cb10 con 0x7f7a240d1d60 2024-04-25T15:22:08.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a29d84640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7a240b8f70 con 0x7f7a240d26b0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a23fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7a241629b0 0x7f7a240b7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52266/0 (socket says 172.21.15.146:52266) 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a23fff640 1 -- 172.21.15.146:0/1388918019 learned_addr learned my addr 172.21.15.146:0/1388918019 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a237fe640 1 --2- 172.21.15.146:0/1388918019 >> v2:172.21.15.177:3300/0 conn(0x7f7a240d1d60 0x7f7a240d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a22ffd640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 0x7f7a24162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278438289 0 0) 0x7f7a240b8cc0 con 0x7f7a241629b0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a0c003520 con 0x7f7a241629b0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2120842320 0 0) 0x7f7a0c003520 con 0x7f7a241629b0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 msgr2=0x7f7a24162470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 0x7f7a24162470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 >> v2:172.21.15.177:3300/0 conn(0x7f7a240d1d60 msgr2=0x7f7a240d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 --2- 172.21.15.146:0/1388918019 >> v2:172.21.15.177:3300/0 conn(0x7f7a240d1d60 0x7f7a240d2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a240b5e80 con 0x7f7a241629b0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f7a1c0019f0 con 0x7f7a241629b0 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 >> v1:172.21.15.146:6789/0 conn(0x7f7a241629b0 legacy=0x7f7a240b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:08.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 0x7f7a0c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a0c003f30 0x7f7a0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a0c004d40 0x7f7a0c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2409cb10 con 0x7f7a0c004d40 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a240b8f70 con 0x7f7a0c003f30 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.305+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7a0c00ba10 con 0x7f7a240d26b0 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a0c004d40 0x7f7a0c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a237fe640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 0x7f7a0c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a22ffd640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a0c003f30 0x7f7a0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 -- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 msgr2=0x7f7a0c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240d26b0 0x7f7a0c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 -- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a0c003f30 msgr2=0x7f7a0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a0c003f30 0x7f7a0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 -- 172.21.15.146:0/1388918019 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0c003520 con 0x7f7a0c004d40 2024-04-25T15:22:08.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.306+0000 7f7a23fff640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a0c004d40 0x7f7a0c005150 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c002230 tx=0x7f7a0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5237d65ed6d7ef65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00a4e0 con 0x7f7a0c004d40 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1c02f070 con 0x7f7a0c004d40 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a227fc640 1 -- 172.21.15.146:0/1388918019 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c02bc90 con 0x7f7a0c004d40 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a29d84640 1 -- 172.21.15.146:0/1388918019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a0c004d40 msgr2=0x7f7a0c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a29d84640 1 --2- 172.21.15.146:0/1388918019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a0c004d40 0x7f7a0c005150 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c002230 tx=0x7f7a0c005830 comp rx=0 tx=0).stop 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a23fff640 1 -- 172.21.15.146:0/1388918019 reap_dead start 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a29d84640 1 -- 172.21.15.146:0/1388918019 shutdown_connections 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a29d84640 1 -- 172.21.15.146:0/1388918019 >> 172.21.15.146:0/1388918019 conn(0x7f7a240b52c0 msgr2=0x7f7a240b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a29d84640 1 -- 172.21.15.146:0/1388918019 shutdown_connections 2024-04-25T15:22:08.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.307+0000 7f7a29d84640 1 -- 172.21.15.146:0/1388918019 wait complete. 2024-04-25T15:22:08.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.309+0000 7f7a29d84640 1 Processor -- start 2024-04-25T15:22:08.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.309+0000 7f7a29d84640 1 -- start start 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240c8a10 0x7f7a240b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 0x7f7a240c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a240b1c20 0x7f7a240b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2409d120 con 0x7f7a2415ee20 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2409cfa0 con 0x7f7a240b1c20 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f7a240d1630 con 0x7f7a240c8a10 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 0x7f7a240c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 0x7f7a240c7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43824/0 (socket says 172.21.15.146:43824) 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 -- 172.21.15.146:0/3158853251 learned_addr learned my addr 172.21.15.146:0/3158853251 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 -- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240c8a10 msgr2=0x7f7a240b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240c8a10 0x7f7a240b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 -- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a240b1c20 msgr2=0x7f7a240b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a23fff640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a240b1c20 0x7f7a240b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a240b1c20 0x7f7a240b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 -- 172.21.15.146:0/3158853251 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a240b2570 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a22ffd640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 0x7f7a240c7270 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7a180118e0 tx=0x7f7a18011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1800faf0 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1800fc90 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a240ca3f0 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18014440 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.311+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a240c7c50 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.312+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2406a0e0 con 0x7f7a2415ee20 2024-04-25T15:22:08.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.313+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f7a18012040 con 0x7f7a2415ee20 2024-04-25T15:22:08.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.313+0000 7f7a0bfff640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a10074540 0x7f7a10076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:08.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.314+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5864+0+0 (secure 0 0 0) 0x7f7a18095990 con 0x7f7a2415ee20 2024-04-25T15:22:08.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.314+0000 7f7a237fe640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a10074540 0x7f7a10076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:08.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.314+0000 7f7a237fe640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a10074540 0x7f7a10076a50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7a240cd250 tx=0x7f7a140095e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:08.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.318+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a240b2570 con 0x7f7a2415ee20 2024-04-25T15:22:08.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:08.498+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7a2415f230 con 0x7f7a2415ee20 2024-04-25T15:22:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.171+0000 7f7a0bfff640 1 -- 172.21.15.146:0/3158853251 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) ==== 169+0+0 (secure 0 0 0) 0x7f7a18019030 con 0x7f7a2415ee20 2024-04-25T15:22:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' removed 2024-04-25T15:22:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a10074540 msgr2=0x7f7a10076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a10074540 0x7f7a10076a50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7a240cd250 tx=0x7f7a140095e0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 msgr2=0x7f7a240c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 0x7f7a240c7270 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7a180118e0 tx=0x7f7a18011da0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 shutdown_connections 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7a10074540 0x7f7a10076a50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a240b1c20 0x7f7a240b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a2415ee20 0x7f7a240c7270 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 --2- 172.21.15.146:0/3158853251 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a240c8a10 0x7f7a240b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.174+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 >> 172.21.15.146:0/3158853251 conn(0x7f7a240b52c0 msgr2=0x7f7a240b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.175+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 shutdown_connections 2024-04-25T15:22:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.175+0000 7f7a29d84640 1 -- 172.21.15.146:0/3158853251 wait complete. 2024-04-25T15:22:09.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-25T15:22:09.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.302+0000 7f103bd72640 1 Processor -- start 2024-04-25T15:22:09.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.302+0000 7f103bd72640 1 -- start start 2024-04-25T15:22:09.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103bd72640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1034070b20 0x7f103406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103bd72640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034162350 0x7f1034157870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103bd72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f103406deb0 con 0x7f103406daa0 2024-04-25T15:22:09.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103bd72640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f103415ed80 con 0x7f1034070b20 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103bd72640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f103415e1f0 con 0x7f1034162350 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103930d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f103406daa0 0x7f1034161c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52278/0 (socket says 172.21.15.146:52278) 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103930d640 1 -- 172.21.15.146:0/3333721239 learned_addr learned my addr 172.21.15.146:0/3333721239 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f1039b0e640 1 --2- 172.21.15.146:0/3333721239 >> v2:172.21.15.177:3300/0 conn(0x7f1034070b20 0x7f103406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f103a30f640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034162350 0x7f1034157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f1038b0c640 1 -- 172.21.15.146:0/3333721239 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033047844 0 0) 0x7f1024001110 con 0x7f103406daa0 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.304+0000 7f1038b0c640 1 -- 172.21.15.146:0/3333721239 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1020003520 con 0x7f103406daa0 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1038b0c640 1 -- 172.21.15.146:0/3333721239 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1797439149 0 0) 0x7f1020003520 con 0x7f103406daa0 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1039b0e640 1 -- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034162350 msgr2=0x7f1034157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1039b0e640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034162350 0x7f1034157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1039b0e640 1 -- 172.21.15.146:0/3333721239 >> v1:172.21.15.146:6789/0 conn(0x7f103406daa0 legacy=0x7f1034161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1039b0e640 1 -- 172.21.15.146:0/3333721239 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103415e450 con 0x7f1034070b20 2024-04-25T15:22:09.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f103a30f640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034162350 0x7f1034157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:09.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1039b0e640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034070b20 0x7f103406d560 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10300097a0 tx=0x7f103002a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e43ab1be01ef572a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.305+0000 7f1038b0c640 1 -- 172.21.15.146:0/3333721239 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030037070 con 0x7f1034070b20 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f1038b0c640 1 -- 172.21.15.146:0/3333721239 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103002ab10 con 0x7f1034070b20 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f1038b0c640 1 -- 172.21.15.146:0/3333721239 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1024001840 con 0x7f1034070b20 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 -- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034070b20 msgr2=0x7f103406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034070b20 0x7f103406d560 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10300097a0 tx=0x7f103002a080 comp rx=0 tx=0).stop 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 -- 172.21.15.146:0/3333721239 shutdown_connections 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034162350 0x7f1034157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 --2- 172.21.15.146:0/3333721239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034070b20 0x7f103406d560 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 -- 172.21.15.146:0/3333721239 >> 172.21.15.146:0/3333721239 conn(0x7f10340d1c10 msgr2=0x7f10340733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 -- 172.21.15.146:0/3333721239 shutdown_connections 2024-04-25T15:22:09.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.306+0000 7f103bd72640 1 -- 172.21.15.146:0/3333721239 wait complete. 2024-04-25T15:22:09.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.308+0000 7f103bd72640 1 Processor -- start 2024-04-25T15:22:09.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.308+0000 7f103bd72640 1 -- start start 2024-04-25T15:22:09.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f103bd72640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 0x7f10340cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f103bd72640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034070b20 0x7f10340b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f103bd72640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034162350 0x7f10340b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f103bd72640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1034070f30 con 0x7f103406daa0 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f103bd72640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f103415ea10 con 0x7f1034162350 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f103bd72640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f10340710b0 con 0x7f1034070b20 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f1039b0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 0x7f10340cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f1039b0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 0x7f10340cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43826/0 (socket says 172.21.15.146:43826) 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.309+0000 7f1039b0e640 1 -- 172.21.15.146:0/841404979 learned_addr learned my addr 172.21.15.146:0/841404979 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f103a30f640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034162350 0x7f10340b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f103930d640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034070b20 0x7f10340b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f1039b0e640 1 -- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034070b20 msgr2=0x7f10340b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f1039b0e640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034070b20 0x7f10340b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f1039b0e640 1 -- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034162350 msgr2=0x7f10340b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f1039b0e640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034162350 0x7f10340b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f1039b0e640 1 -- 172.21.15.146:0/841404979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10340cb7c0 con 0x7f103406daa0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f1039b0e640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 0x7f10340cd0c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f10300098c0 tx=0x7f10300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030041020 con 0x7f103406daa0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103003d620 con 0x7f103406daa0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10340cbaa0 con 0x7f103406daa0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030037040 con 0x7f103406daa0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.310+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10340cc060 con 0x7f103406daa0 2024-04-25T15:22:09.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.311+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f103406a0e0 con 0x7f103406daa0 2024-04-25T15:22:09.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.312+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f1030044020 con 0x7f103406daa0 2024-04-25T15:22:09.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.312+0000 7f102a7fc640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1018074540 0x7f1018076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.312+0000 7f103930d640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1018074540 0x7f1018076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.312+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f10300b56a0 con 0x7f103406daa0 2024-04-25T15:22:09.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.313+0000 7f103930d640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1018074540 0x7f1018076a50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f10240013b0 tx=0x7f1024009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.317+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f103004b030 con 0x7f103406daa0 2024-04-25T15:22:09.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.494+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7f10340afe00 con 0x7f103406daa0 2024-04-25T15:22:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.497+0000 7f102a7fc640 1 -- 172.21.15.146:0/841404979 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7f103007ff60 con 0x7f103406daa0 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1018074540 msgr2=0x7f1018076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1018074540 0x7f1018076a50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f10240013b0 tx=0x7f1024009040 comp rx=0 tx=0).stop 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 msgr2=0x7f10340cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 0x7f10340cd0c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f10300098c0 tx=0x7f10300046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 shutdown_connections 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1018074540 0x7f1018076a50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1034162350 0x7f10340b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1034070b20 0x7f10340b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 --2- 172.21.15.146:0/841404979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f103406daa0 0x7f10340cd0c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 >> 172.21.15.146:0/841404979 conn(0x7f10340d1c10 msgr2=0x7f1034161c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 shutdown_connections 2024-04-25T15:22:09.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.500+0000 7f103bd72640 1 -- 172.21.15.146:0/841404979 wait complete. 2024-04-25T15:22:09.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:22:09.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.622+0000 7f6069ed8640 1 Processor -- start 2024-04-25T15:22:09.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.622+0000 7f6069ed8640 1 -- start start 2024-04-25T15:22:09.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.623+0000 7f6069ed8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f60640b2cd0 0x7f60640d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.623+0000 7f6069ed8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60640d3300 0x7f6064071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6069ed8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6064071f70 con 0x7f6064071b60 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6069ed8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f60640d2260 con 0x7f60640d3300 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6069ed8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f60640d2540 con 0x7f60640b2cd0 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60640d3300 0x7f6064071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60640d3300 0x7f6064071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43828/0 (socket says 172.21.15.146:43828) 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 -- 172.21.15.146:0/3861727523 learned_addr learned my addr 172.21.15.146:0/3861727523 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f60637fe640 1 --2- 172.21.15.146:0/3861727523 >> v2:172.21.15.177:3300/0 conn(0x7f60640b2cd0 0x7f60640d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 -- 172.21.15.146:0/3861727523 >> v2:172.21.15.177:3300/0 conn(0x7f60640b2cd0 msgr2=0x7f60640d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 --2- 172.21.15.146:0/3861727523 >> v2:172.21.15.177:3300/0 conn(0x7f60640b2cd0 0x7f60640d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 -- 172.21.15.146:0/3861727523 >> v1:172.21.15.177:6789/0 conn(0x7f6064071b60 legacy=0x7f606408cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f6062ffd640 1 -- 172.21.15.146:0/3861727523 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60640d2820 con 0x7f60640d3300 2024-04-25T15:22:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.624+0000 7f60637fe640 1 --2- 172.21.15.146:0/3861727523 >> v2:172.21.15.177:3300/0 conn(0x7f60640b2cd0 0x7f60640d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:09.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.625+0000 7f6062ffd640 1 --2- 172.21.15.146:0/3861727523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60640d3300 0x7f6064071620 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f606408c9d0 tx=0x7f605802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec052df3e006e8c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.625+0000 7f60627fc640 1 -- 172.21.15.146:0/3861727523 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058037070 con 0x7f60640d3300 2024-04-25T15:22:09.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.625+0000 7f60627fc640 1 -- 172.21.15.146:0/3861727523 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6058032440 con 0x7f60640d3300 2024-04-25T15:22:09.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.625+0000 7f60627fc640 1 -- 172.21.15.146:0/3861727523 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058032810 con 0x7f60640d3300 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.625+0000 7f6069ed8640 1 -- 172.21.15.146:0/3861727523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60640d3300 msgr2=0x7f6064071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 --2- 172.21.15.146:0/3861727523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60640d3300 0x7f6064071620 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f606408c9d0 tx=0x7f605802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 -- 172.21.15.146:0/3861727523 shutdown_connections 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 --2- 172.21.15.146:0/3861727523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60640d3300 0x7f6064071620 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 --2- 172.21.15.146:0/3861727523 >> v2:172.21.15.177:3300/0 conn(0x7f60640b2cd0 0x7f60640d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 -- 172.21.15.146:0/3861727523 >> 172.21.15.146:0/3861727523 conn(0x7f60640b52e0 msgr2=0x7f60640b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 -- 172.21.15.146:0/3861727523 shutdown_connections 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.626+0000 7f6069ed8640 1 -- 172.21.15.146:0/3861727523 wait complete. 2024-04-25T15:22:09.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.627+0000 7f6069ed8640 1 Processor -- start 2024-04-25T15:22:09.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.628+0000 7f6069ed8640 1 -- start start 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6069ed8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 0x7f6064161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6069ed8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640b2cd0 0x7f606415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6069ed8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640d3300 0x7f60640b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6069ed8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f60640cfb70 con 0x7f6064071b60 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6069ed8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f60640d2de0 con 0x7f60640b2cd0 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6069ed8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f606408dbf0 con 0x7f60640d3300 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 0x7f6064161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 0x7f6064161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43830/0 (socket says 172.21.15.146:43830) 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 -- 172.21.15.146:0/335345542 learned_addr learned my addr 172.21.15.146:0/335345542 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6062ffd640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640b2cd0 0x7f606415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 -- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640d3300 msgr2=0x7f60640b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6063fff640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640d3300 0x7f60640b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640d3300 0x7f60640b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 -- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640b2cd0 msgr2=0x7f606415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640b2cd0 0x7f606415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 -- 172.21.15.146:0/335345542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60640b2aa0 con 0x7f6064071b60 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6062ffd640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640b2cd0 0x7f606415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f6063fff640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640d3300 0x7f60640b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.629+0000 7f60637fe640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 0x7f6064161030 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f605400c7f0 tx=0x7f605400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.630+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054005b90 con 0x7f6064071b60 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.630+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6054005d30 con 0x7f6064071b60 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.630+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60640ca0a0 con 0x7f6064071b60 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.630+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054006270 con 0x7f6064071b60 2024-04-25T15:22:09.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.630+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6064160810 con 0x7f6064071b60 2024-04-25T15:22:09.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.631+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606406a0e0 con 0x7f6064071b60 2024-04-25T15:22:09.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.631+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f6054006620 con 0x7f6064071b60 2024-04-25T15:22:09.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.632+0000 7f604bfff640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6050074570 0x7f6050076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.632+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f60540952a0 con 0x7f6064071b60 2024-04-25T15:22:09.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.632+0000 7f6062ffd640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6050074570 0x7f6050076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.632+0000 7f6062ffd640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6050074570 0x7f6050076a80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6058034000 tx=0x7f605802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.636+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6054015020 con 0x7f6064071b60 2024-04-25T15:22:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.811+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f60640cfe10 con 0x7f6064071b60 2024-04-25T15:22:09.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.815+0000 7f604bfff640 1 -- 172.21.15.146:0/335345542 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7f605405fb70 con 0x7f6064071b60 2024-04-25T15:22:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6050074570 msgr2=0x7f6050076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6050074570 0x7f6050076a80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6058034000 tx=0x7f605802e000 comp rx=0 tx=0).stop 2024-04-25T15:22:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 msgr2=0x7f6064161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 0x7f6064161030 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f605400c7f0 tx=0x7f605400ccb0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 shutdown_connections 2024-04-25T15:22:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6050074570 0x7f6050076a80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640d3300 0x7f60640b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640b2cd0 0x7f606415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 --2- 172.21.15.146:0/335345542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6064071b60 0x7f6064161030 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 >> 172.21.15.146:0/335345542 conn(0x7f60640b52e0 msgr2=0x7f60640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 shutdown_connections 2024-04-25T15:22:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.817+0000 7f6069ed8640 1 -- 172.21.15.146:0/335345542 wait complete. 2024-04-25T15:22:09.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_cache_quiesce_splitauth 2024-04-25T15:22:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.939+0000 7fa08c182640 1 Processor -- start 2024-04-25T15:22:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.939+0000 7fa08c182640 1 -- start start 2024-04-25T15:22:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08c182640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0840cf0f0 0x7fa0840d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08c182640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0840cf500 con 0x7fa084071c70 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08c182640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0840cf7d0 con 0x7fa0840b2cd0 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08c182640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa08408da90 con 0x7fa0840cf0f0 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08a71f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa084071c70 0x7fa084072080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52280/0 (socket says 172.21.15.146:52280) 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08a71f640 1 -- 172.21.15.146:0/3302010078 learned_addr learned my addr 172.21.15.146:0/3302010078 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096101419 0 0) 0x7fa0840cf500 con 0x7fa084071c70 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08971d640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0840cf0f0 0x7fa0840d2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa070003520 con 0x7fa084071c70 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1043571620 0 0) 0x7fa070003520 con 0x7fa084071c70 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0840cf0f0 msgr2=0x7fa0840d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0840cf0f0 0x7fa0840d2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 >> v1:172.21.15.177:6789/0 conn(0x7fa0840b2cd0 legacy=0x7fa0840d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0840cfab0 con 0x7fa084071c70 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fa07c0019f0 con 0x7fa084071c70 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 >> v1:172.21.15.146:6789/0 conn(0x7fa084071c70 legacy=0x7fa084072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0700039e0 0x7fa070003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070004800 0x7fa070004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa07000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa07000c1d0 con 0x7fa070005350 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa07000c380 con 0x7fa070004800 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa07000c580 con 0x7fa0700039e0 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.941+0000 7fa08971d640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0840cf0f0 0x7fa0840d2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08a71f640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa07000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa089f1e640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0700039e0 0x7fa070003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08971d640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070004800 0x7fa070004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08a71f640 1 -- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0700039e0 msgr2=0x7fa070003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08a71f640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa0700039e0 0x7fa070003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08a71f640 1 -- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070004800 msgr2=0x7fa070004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08a71f640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070004800 0x7fa070004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.942+0000 7fa08a71f640 1 -- 172.21.15.146:0/3302010078 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa070003520 con 0x7fa070005350 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa08a71f640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa07000b9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa0840ca870 tx=0x7fa070005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1043de7b6e77d489 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c00a4a0 con 0x7fa070005350 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07c02f070 con 0x7fa070005350 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa088f1c640 1 -- 172.21.15.146:0/3302010078 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c02bcf0 con 0x7fa070005350 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa08c182640 1 -- 172.21.15.146:0/3302010078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 msgr2=0x7fa07000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa08c182640 1 --2- 172.21.15.146:0/3302010078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa07000b9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa0840ca870 tx=0x7fa070005830 comp rx=0 tx=0).stop 2024-04-25T15:22:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa08a71f640 1 -- 172.21.15.146:0/3302010078 reap_dead start 2024-04-25T15:22:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa08c182640 1 -- 172.21.15.146:0/3302010078 shutdown_connections 2024-04-25T15:22:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.943+0000 7fa08c182640 1 -- 172.21.15.146:0/3302010078 >> 172.21.15.146:0/3302010078 conn(0x7fa0840b52e0 msgr2=0x7fa0840b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.944+0000 7fa08c182640 1 -- 172.21.15.146:0/3302010078 shutdown_connections 2024-04-25T15:22:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.944+0000 7fa08c182640 1 -- 172.21.15.146:0/3302010078 wait complete. 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.945+0000 7fa08c182640 1 Processor -- start 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.945+0000 7fa08c182640 1 -- start start 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa08415e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08406e8d0 0x7fa0841f9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0841f9ab0 0x7fa084160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0840717e0 con 0x7fa070005350 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa0840d0070 con 0x7fa0841f9ab0 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa0840d2030 con 0x7fa08406e8d0 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa08415e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa08415e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43840/0 (socket says 172.21.15.146:43840) 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 -- 172.21.15.146:0/1130785614 learned_addr learned my addr 172.21.15.146:0/1130785614 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08a71f640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0841f9ab0 0x7fa084160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08971d640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08406e8d0 0x7fa0841f9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 -- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08406e8d0 msgr2=0x7fa0841f9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08406e8d0 0x7fa0841f9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 -- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0841f9ab0 msgr2=0x7fa084160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0841f9ab0 0x7fa084160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 -- 172.21.15.146:0/1130785614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0841f9ec0 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08971d640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08406e8d0 0x7fa0841f9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa089f1e640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa08415e780 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa08000f610 tx=0x7fa08000fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0800052e0 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa080018070 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08406ee10 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080003640 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.947+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08406f3d0 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.948+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08406a0e0 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.949+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa0800037e0 con 0x7fa070005350 2024-04-25T15:22:09.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.950+0000 7fa07a7fc640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa068074570 0x7fa068076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:09.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.950+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fa080094f10 con 0x7fa070005350 2024-04-25T15:22:09.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.950+0000 7fa08971d640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa068074570 0x7fa068076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:09.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.950+0000 7fa08971d640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa068074570 0x7fa068076a80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa0840a13a0 tx=0x7fa074008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:09.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:09.954+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0841f9ec0 con 0x7fa070005350 2024-04-25T15:22:10.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.134+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"} v 0) -- 0x7fa084072bc0 con 0x7fa070005350 2024-04-25T15:22:10.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.138+0000 7fa07a7fc640 1 -- 172.21.15.146:0/1130785614 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"}]=0 v15) ==== 110+0+0 (secure 0 0 0) 0x7fa08005f8f0 con 0x7fa070005350 2024-04-25T15:22:10.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa068074570 msgr2=0x7fa068076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:10.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa068074570 0x7fa068076a80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa0840a13a0 tx=0x7fa074008040 comp rx=0 tx=0).stop 2024-04-25T15:22:10.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 msgr2=0x7fa08415e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa08415e780 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa08000f610 tx=0x7fa08000fad0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 shutdown_connections 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa068074570 0x7fa068076a80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0841f9ab0 0x7fa084160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08406e8d0 0x7fa0841f9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 --2- 172.21.15.146:0/1130785614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa070005350 0x7fa08415e780 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.140+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 >> 172.21.15.146:0/1130785614 conn(0x7fa0840b52e0 msgr2=0x7fa0840b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.141+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 shutdown_connections 2024-04-25T15:22:10.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.141+0000 7fa08c182640 1 -- 172.21.15.146:0/1130785614 wait complete. 2024-04-25T15:22:10.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-25T15:22:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.260+0000 7fcf178b8640 1 Processor -- start 2024-04-25T15:22:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.261+0000 7fcf178b8640 1 -- start start 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf178b8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 0x7fcf10071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf178b8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcf100cf040 0x7fcf100cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf178b8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf100d2460 con 0x7fcf1008cc60 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf178b8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcf100d2a20 con 0x7fcf100cf040 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf178b8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcf100d2740 con 0x7fcf100b8dc0 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf15e55640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcf1008cc60 0x7fcf1008d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52294/0 (socket says 172.21.15.146:52294) 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf15e55640 1 -- 172.21.15.146:0/802352310 learned_addr learned my addr 172.21.15.146:0/802352310 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf14e53640 1 --2- 172.21.15.146:0/802352310 >> v2:172.21.15.177:3300/0 conn(0x7fcf100cf040 0x7fcf100cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695765897 0 0) 0x7fcf100d2460 con 0x7fcf1008cc60 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.262+0000 7fcf15654640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 0x7fcf10071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcefc003520 con 0x7fcf1008cc60 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3176958964 0 0) 0x7fcefc003520 con 0x7fcf1008cc60 2024-04-25T15:22:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 msgr2=0x7fcf10071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 0x7fcf10071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 >> v2:172.21.15.177:3300/0 conn(0x7fcf100cf040 msgr2=0x7fcf100cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 --2- 172.21.15.146:0/802352310 >> v2:172.21.15.177:3300/0 conn(0x7fcf100cf040 0x7fcf100cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf100ce840 con 0x7fcf1008cc60 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fcf08001670 con 0x7fcf1008cc60 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 >> v1:172.21.15.146:6789/0 conn(0x7fcf1008cc60 legacy=0x7fcf1008d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 0x7fcefc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcefc003f30 0x7fcefc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcefc004d40 0x7fcefc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcf100d2740 con 0x7fcefc003f30 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc00ba70 con 0x7fcefc004d40 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcefc00bc40 con 0x7fcf100b8dc0 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf14e53640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcefc003f30 0x7fcefc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf15654640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 0x7fcefc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf15e55640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcefc004d40 0x7fcefc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf14e53640 1 -- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 msgr2=0x7fcefc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf14e53640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf100b8dc0 0x7fcefc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf14e53640 1 -- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcefc004d40 msgr2=0x7fcefc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf14e53640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcefc004d40 0x7fcefc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.263+0000 7fcf14e53640 1 -- 172.21.15.146:0/802352310 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc003520 con 0x7fcefc003f30 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.264+0000 7fcf14e53640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcefc003f30 0x7fcefc004800 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fcf100749d0 tx=0x7fcefc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72039200c8c0747c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.264+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf00036070 con 0x7fcefc003f30 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.264+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf00031400 con 0x7fcefc003f30 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.264+0000 7fcf07fff640 1 -- 172.21.15.146:0/802352310 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf00009690 con 0x7fcefc003f30 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf178b8640 1 -- 172.21.15.146:0/802352310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcefc003f30 msgr2=0x7fcefc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf178b8640 1 --2- 172.21.15.146:0/802352310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcefc003f30 0x7fcefc004800 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fcf100749d0 tx=0x7fcefc005830 comp rx=0 tx=0).stop 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf15e55640 1 -- 172.21.15.146:0/802352310 reap_dead start 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf178b8640 1 -- 172.21.15.146:0/802352310 shutdown_connections 2024-04-25T15:22:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf178b8640 1 -- 172.21.15.146:0/802352310 >> 172.21.15.146:0/802352310 conn(0x7fcf100b5360 msgr2=0x7fcf100b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:10.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf178b8640 1 -- 172.21.15.146:0/802352310 shutdown_connections 2024-04-25T15:22:10.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.265+0000 7fcf178b8640 1 -- 172.21.15.146:0/802352310 wait complete. 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.267+0000 7fcf178b8640 1 Processor -- start 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.267+0000 7fcf178b8640 1 -- start start 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf178b8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcefc003f30 0x7fcf10160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf178b8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf1014eae0 0x7fcf1015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf178b8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 0x7fcf1015aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf178b8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcf100b2e80 con 0x7fcf1014eef0 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf178b8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf100d20f0 con 0x7fcf1014eae0 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf178b8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcf100b3000 con 0x7fcefc003f30 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 0x7fcf1015aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 0x7fcf1015aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43850/0 (socket says 172.21.15.146:43850) 2024-04-25T15:22:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 -- 172.21.15.146:0/1956424779 learned_addr learned my addr 172.21.15.146:0/1956424779 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15654640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcefc003f30 0x7fcf10160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf14e53640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf1014eae0 0x7fcf1015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 -- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcefc003f30 msgr2=0x7fcf10160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcefc003f30 0x7fcf10160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 -- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf1014eae0 msgr2=0x7fcf1015a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf1014eae0 0x7fcf1015a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.268+0000 7fcf15e55640 1 -- 172.21.15.146:0/1956424779 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1015f150 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.269+0000 7fcf15e55640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 0x7fcf1015aed0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcf080011e0 tx=0x7fcf08007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.269+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0800f600 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.269+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf08018070 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.269+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0800fa00 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.269+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf100b06a0 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.269+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf100b0c60 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.270+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf10161260 con 0x7fcf1014eef0 2024-04-25T15:22:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.270+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fcf0800fba0 con 0x7fcf1014eef0 2024-04-25T15:22:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.271+0000 7fcf05ffb640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcef4074540 0x7fcef4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.271+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fcf08094da0 con 0x7fcf1014eef0 2024-04-25T15:22:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.271+0000 7fcf15654640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcef4074540 0x7fcef4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.271+0000 7fcf15654640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcef4074540 0x7fcef4076a50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcf100d3020 tx=0x7fcf0c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.275+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf0805f130 con 0x7fcf1014eef0 2024-04-25T15:22:10.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:10.530+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7fcf1006a0e0 con 0x7fcf1014eef0 2024-04-25T15:22:11.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.180+0000 7fcf05ffb640 1 -- 172.21.15.146:0/1956424779 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v157) ==== 137+0+0 (secure 0 0 0) 0x7fcf08063430 con 0x7fcf1014eef0 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcef4074540 msgr2=0x7fcef4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcef4074540 0x7fcef4076a50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcf100d3020 tx=0x7fcf0c00b040 comp rx=0 tx=0).stop 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 msgr2=0x7fcf1015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 0x7fcf1015aed0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fcf080011e0 tx=0x7fcf08007430 comp rx=0 tx=0).stop 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 shutdown_connections 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcef4074540 0x7fcef4076a50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf1014eef0 0x7fcf1015aed0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf1014eae0 0x7fcf1015a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 --2- 172.21.15.146:0/1956424779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcefc003f30 0x7fcf10160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 >> 172.21.15.146:0/1956424779 conn(0x7fcf100b5360 msgr2=0x7fcf1015dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 shutdown_connections 2024-04-25T15:22:11.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.183+0000 7fcf178b8640 1 -- 172.21.15.146:0/1956424779 wait complete. 2024-04-25T15:22:11.196 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. ... ok 2024-04-25T15:22:11.196 INFO:tasks.cephfs_test_runner:Starting test: test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce) That quiesce completes with fragmentation in the background. 2024-04-25T15:22:11.196 INFO:tasks.cephfs_test_runner:test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-25T15:22:11.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.305+0000 7ff9a96b6640 1 Processor -- start 2024-04-25T15:22:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.305+0000 7ff9a96b6640 1 -- start start 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.306+0000 7ff9a96b6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9a40d4f60 0x7ff9a40d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.306+0000 7ff9a96b6640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9a406ccc0 0x7ff9a406d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a96b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a40b9200 con 0x7ff9a40d0020 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a96b6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a406c550 con 0x7ff9a40d4f60 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a96b6640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a406c830 con 0x7ff9a406ccc0 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a2ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9a40d4f60 0x7ff9a40d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a27fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff9a40d0020 0x7ff9a40d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38102/0 (socket says 172.21.15.146:38102) 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a27fc640 1 -- 172.21.15.146:0/337825445 learned_addr learned my addr 172.21.15.146:0/337825445 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a37fe640 1 --2- 172.21.15.146:0/337825445 >> v2:172.21.15.177:3300/0 conn(0x7ff9a406ccc0 0x7ff9a406d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/337825445 >> v2:172.21.15.177:3300/0 conn(0x7ff9a406ccc0 msgr2=0x7ff9a406d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a2ffd640 1 --2- 172.21.15.146:0/337825445 >> v2:172.21.15.177:3300/0 conn(0x7ff9a406ccc0 0x7ff9a406d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/337825445 >> v1:172.21.15.146:6789/0 conn(0x7ff9a40d0020 legacy=0x7ff9a40d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a1ffb640 1 -- 172.21.15.146:0/337825445 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3532873677 0 0) 0x7ff998001110 con 0x7ff9a40d0020 2024-04-25T15:22:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/337825445 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a40d1070 con 0x7ff9a40d4f60 2024-04-25T15:22:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.307+0000 7ff9a37fe640 1 --2- 172.21.15.146:0/337825445 >> v2:172.21.15.177:3300/0 conn(0x7ff9a406ccc0 0x7ff9a406d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a2ffd640 1 --2- 172.21.15.146:0/337825445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a40d4f60 0x7ff9a40d5370 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40d3e10 tx=0x7ff99402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa73301d56eed410 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a1ffb640 1 -- 172.21.15.146:0/337825445 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994037070 con 0x7ff9a40d4f60 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a1ffb640 1 -- 172.21.15.146:0/337825445 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff99402add0 con 0x7ff9a40d4f60 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a1ffb640 1 -- 172.21.15.146:0/337825445 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994032660 con 0x7ff9a40d4f60 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a96b6640 1 -- 172.21.15.146:0/337825445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a40d4f60 msgr2=0x7ff9a40d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/337825445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a40d4f60 0x7ff9a40d5370 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40d3e10 tx=0x7ff99402a080 comp rx=0 tx=0).stop 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a96b6640 1 -- 172.21.15.146:0/337825445 shutdown_connections 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/337825445 >> v2:172.21.15.177:3300/0 conn(0x7ff9a406ccc0 0x7ff9a406d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/337825445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a40d4f60 0x7ff9a40d5370 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.308+0000 7ff9a96b6640 1 -- 172.21.15.146:0/337825445 >> 172.21.15.146:0/337825445 conn(0x7ff9a40d2450 msgr2=0x7ff9a40d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.309+0000 7ff9a96b6640 1 -- 172.21.15.146:0/337825445 shutdown_connections 2024-04-25T15:22:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.309+0000 7ff9a96b6640 1 -- 172.21.15.146:0/337825445 wait complete. 2024-04-25T15:22:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.310+0000 7ff9a96b6640 1 Processor -- start 2024-04-25T15:22:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.310+0000 7ff9a96b6640 1 -- start start 2024-04-25T15:22:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 0x7ff9a415ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a40d0020 0x7ff9a41f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a40d4f60 0x7ff9a41f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a40b5150 con 0x7ff9a406ccc0 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a40d15f0 con 0x7ff9a40d0020 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a40b52d0 con 0x7ff9a40d4f60 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 0x7ff9a415ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 0x7ff9a415ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57926/0 (socket says 172.21.15.146:57926) 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/841891520 learned_addr learned my addr 172.21.15.146:0/841891520 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a27fc640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a40d0020 0x7ff9a41f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a37fe640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a40d4f60 0x7ff9a41f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a40d4f60 msgr2=0x7ff9a41f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a40d4f60 0x7ff9a41f6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a40d0020 msgr2=0x7ff9a41f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a40d0020 0x7ff9a41f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 -- 172.21.15.146:0/841891520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a4157670 con 0x7ff9a406ccc0 2024-04-25T15:22:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a2ffd640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 0x7ff9a415ce60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff994000c00 tx=0x7ff9940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994037040 con 0x7ff9a406ccc0 2024-04-25T15:22:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff994042070 con 0x7ff9a406ccc0 2024-04-25T15:22:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a4157950 con 0x7ff9a406ccc0 2024-04-25T15:22:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.312+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99403b890 con 0x7ff9a406ccc0 2024-04-25T15:22:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.313+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a4159fd0 con 0x7ff9a406ccc0 2024-04-25T15:22:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.313+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a406a0e0 con 0x7ff9a406ccc0 2024-04-25T15:22:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.314+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7ff99402ac50 con 0x7ff9a406ccc0 2024-04-25T15:22:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.315+0000 7ff98b7fe640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff990074540 0x7ff990076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.315+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7ff9940b54f0 con 0x7ff9a406ccc0 2024-04-25T15:22:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.315+0000 7ff9a27fc640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff990074540 0x7ff990076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.315+0000 7ff9a27fc640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff990074540 0x7ff990076a50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4156a00 tx=0x7ff998007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.319+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff99404c020 con 0x7ff9a406ccc0 2024-04-25T15:22:11.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.494+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9a40b5a70 con 0x7ff9a406ccc0 2024-04-25T15:22:11.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.495+0000 7ff98b7fe640 1 -- 172.21.15.146:0/841891520 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+13471 (secure 0 0 0) 0x7ff99407f880 con 0x7ff9a406ccc0 2024-04-25T15:22:11.495 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:11.495 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":45,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:09.169969+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6834/377672985":"2024-04-26T15:22:06.165092+0000","172.21.15.177:6835/539736758":"2024-04-26T15:21:49.486202+0000","172.21.15.177:6832/377672985":"2024-04-26T15:22:06.165092+0000","172.21.15.177:6833/539736758":"2024-04-26T15:21:49.486202+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.496+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff990074540 msgr2=0x7ff990076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.496+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff990074540 0x7ff990076a50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4156a00 tx=0x7ff998007de0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.496+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 msgr2=0x7ff9a415ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.496+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 0x7ff9a415ce60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff994000c00 tx=0x7ff9940046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 shutdown_connections 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff990074540 0x7ff990076a50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9a40d4f60 0x7ff9a41f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9a40d0020 0x7ff9a41f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 --2- 172.21.15.146:0/841891520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a406ccc0 0x7ff9a415ce60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 >> 172.21.15.146:0/841891520 conn(0x7ff9a40d2450 msgr2=0x7ff9a40ba600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 shutdown_connections 2024-04-25T15:22:11.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.497+0000 7ff9a96b6640 1 -- 172.21.15.146:0/841891520 wait complete. 2024-04-25T15:22:11.509 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:11.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:11.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.619+0000 7f53bcbce640 1 Processor -- start 2024-04-25T15:22:11.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.619+0000 7f53bcbce640 1 -- start start 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53bcbce640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f53b815e070 0x7f53b80cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53bcbce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53b809bd80 0x7f53b80ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53bcbce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b80d2680 0x7f53b80d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53bcbce640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f53b80d2fd0 con 0x7f53b809bd80 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53bcbce640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f53b80ce920 con 0x7f53b815e070 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53bcbce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f53b80cec00 con 0x7f53b80d2680 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53b809bd80 0x7f53b80ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53b809bd80 0x7f53b80ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57938/0 (socket says 172.21.15.146:57938) 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 -- 172.21.15.146:0/1222047310 learned_addr learned my addr 172.21.15.146:0/1222047310 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b659c640 1 --2- 172.21.15.146:0/1222047310 >> v2:172.21.15.177:3300/0 conn(0x7f53b815e070 0x7f53b80cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b6d9d640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b80d2680 0x7f53b80d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 -- 172.21.15.146:0/1222047310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b80d2680 msgr2=0x7f53b80d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b80d2680 0x7f53b80d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 -- 172.21.15.146:0/1222047310 >> v2:172.21.15.177:3300/0 conn(0x7f53b815e070 msgr2=0x7f53b80cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 --2- 172.21.15.146:0/1222047310 >> v2:172.21.15.177:3300/0 conn(0x7f53b815e070 0x7f53b80cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b5d9b640 1 -- 172.21.15.146:0/1222047310 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b80ceee0 con 0x7f53b809bd80 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.621+0000 7f53b6d9d640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b80d2680 0x7f53b80d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.622+0000 7f53b5d9b640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b809bd80 0x7f53b80ce060 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f53b8072f80 tx=0x7f53ac02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570aa033990e0944 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.622+0000 7f53b559a640 1 -- 172.21.15.146:0/1222047310 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac037070 con 0x7f53b809bd80 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.622+0000 7f53b559a640 1 -- 172.21.15.146:0/1222047310 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53ac032440 con 0x7f53b809bd80 2024-04-25T15:22:11.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.622+0000 7f53b559a640 1 -- 172.21.15.146:0/1222047310 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac032810 con 0x7f53b809bd80 2024-04-25T15:22:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 -- 172.21.15.146:0/1222047310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b809bd80 msgr2=0x7f53b80ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b809bd80 0x7f53b80ce060 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f53b8072f80 tx=0x7f53ac02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 -- 172.21.15.146:0/1222047310 shutdown_connections 2024-04-25T15:22:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b80d2680 0x7f53b80d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 --2- 172.21.15.146:0/1222047310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b809bd80 0x7f53b80ce060 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 --2- 172.21.15.146:0/1222047310 >> v2:172.21.15.177:3300/0 conn(0x7f53b815e070 0x7f53b80cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 -- 172.21.15.146:0/1222047310 >> 172.21.15.146:0/1222047310 conn(0x7f53b80b7400 msgr2=0x7f53b80b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 -- 172.21.15.146:0/1222047310 shutdown_connections 2024-04-25T15:22:11.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.623+0000 7f53bcbce640 1 -- 172.21.15.146:0/1222047310 wait complete. 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.625+0000 7f53bcbce640 1 Processor -- start 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.625+0000 7f53bcbce640 1 -- start start 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53bcbce640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b809bd80 0x7f53b8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53bcbce640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 0x7f53b80b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53bcbce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b815e070 0x7f53b80cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53bcbce640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f53b80cff00 con 0x7f53b80d2680 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53bcbce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53b80cf4a0 con 0x7f53b815e070 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53bcbce640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f53b80ce5a0 con 0x7f53b809bd80 2024-04-25T15:22:11.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53b5d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 0x7f53b80b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53b5d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 0x7f53b80b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57948/0 (socket says 172.21.15.146:57948) 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53b5d9b640 1 -- 172.21.15.146:0/2407085219 learned_addr learned my addr 172.21.15.146:0/2407085219 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53b6d9d640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b815e070 0x7f53b80cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.626+0000 7f53b659c640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b809bd80 0x7f53b8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53b5d9b640 1 -- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b809bd80 msgr2=0x7f53b8157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53b5d9b640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b809bd80 0x7f53b8157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53b5d9b640 1 -- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b815e070 msgr2=0x7f53b80cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53b5d9b640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b815e070 0x7f53b80cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53b5d9b640 1 -- 172.21.15.146:0/2407085219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b80b4200 con 0x7f53b80d2680 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53b5d9b640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 0x7f53b80b3490 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f53ac02a220 tx=0x7f53ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac041020 con 0x7f53b80d2680 2024-04-25T15:22:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53ac0306c0 con 0x7f53b80d2680 2024-04-25T15:22:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b80b2230 con 0x7f53b80d2680 2024-04-25T15:22:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac037040 con 0x7f53b80d2680 2024-04-25T15:22:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.627+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b80b27f0 con 0x7f53b80d2680 2024-04-25T15:22:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.628+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b806a0e0 con 0x7f53b80d2680 2024-04-25T15:22:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.629+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f53ac044020 con 0x7f53b80d2680 2024-04-25T15:22:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.629+0000 7f53a6ffd640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5398074540 0x7f5398076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.629+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f53ac0b58d0 con 0x7f53b80d2680 2024-04-25T15:22:11.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.629+0000 7f53b659c640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5398074540 0x7f5398076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.630+0000 7f53b659c640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5398074540 0x7f5398076a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f53a8003b80 tx=0x7f53a8009880 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.634+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53ac04b030 con 0x7f53b80d2680 2024-04-25T15:22:11.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.807+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53b806ebf0 con 0x7f53b80d2680 2024-04-25T15:22:11.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.808+0000 7f53a6ffd640 1 -- 172.21.15.146:0/2407085219 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7f53ac088190 con 0x7f53b80d2680 2024-04-25T15:22:11.808 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5398074540 msgr2=0x7f5398076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5398074540 0x7f5398076a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f53a8003b80 tx=0x7f53a8009880 comp rx=0 tx=0).stop 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 msgr2=0x7f53b80b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 0x7f53b80b3490 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f53ac02a220 tx=0x7f53ac0046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 shutdown_connections 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5398074540 0x7f5398076a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b815e070 0x7f53b80cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53b80d2680 0x7f53b80b3490 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 --2- 172.21.15.146:0/2407085219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53b809bd80 0x7f53b8157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 >> 172.21.15.146:0/2407085219 conn(0x7f53b80b7400 msgr2=0x7f53b81591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 shutdown_connections 2024-04-25T15:22:11.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.810+0000 7f53bcbce640 1 -- 172.21.15.146:0/2407085219 wait complete. 2024-04-25T15:22:11.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_dir_fragment' 2024-04-25T15:22:11.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.930+0000 7f3251a92640 1 Processor -- start 2024-04-25T15:22:11.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.930+0000 7f3251a92640 1 -- start start 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.931+0000 7f3251a92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c06ee60 0x7f324c0b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.931+0000 7f3251a92640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f324c163eb0 0x7f324c1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.931+0000 7f3251a92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c164800 0x7f324c160830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.931+0000 7f3251a92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f324c08d730 con 0x7f324c164800 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.931+0000 7f3251a92640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f324c06f440 con 0x7f324c163eb0 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.931+0000 7f3251a92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f324c0b8eb0 con 0x7f324c06ee60 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c164800 0x7f324c160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c164800 0x7f324c160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57950/0 (socket says 172.21.15.146:57950) 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 -- 172.21.15.146:0/2450807692 learned_addr learned my addr 172.21.15.146:0/2450807692 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:11.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324affd640 1 --2- 172.21.15.146:0/2450807692 >> v2:172.21.15.177:3300/0 conn(0x7f324c163eb0 0x7f324c1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324b7fe640 1 --2- 172.21.15.146:0/2450807692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c06ee60 0x7f324c0b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 -- 172.21.15.146:0/2450807692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c06ee60 msgr2=0x7f324c0b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 --2- 172.21.15.146:0/2450807692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c06ee60 0x7f324c0b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 -- 172.21.15.146:0/2450807692 >> v2:172.21.15.177:3300/0 conn(0x7f324c163eb0 msgr2=0x7f324c1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 --2- 172.21.15.146:0/2450807692 >> v2:172.21.15.177:3300/0 conn(0x7f324c163eb0 0x7f324c1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.932+0000 7f324bfff640 1 -- 172.21.15.146:0/2450807692 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c08ce00 con 0x7f324c164800 2024-04-25T15:22:11.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.933+0000 7f324bfff640 1 --2- 172.21.15.146:0/2450807692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c164800 0x7f324c160830 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f324c0710e0 tx=0x7f324402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f5a295a3364372 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.933+0000 7f324a7fc640 1 -- 172.21.15.146:0/2450807692 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244036070 con 0x7f324c164800 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.933+0000 7f324a7fc640 1 -- 172.21.15.146:0/2450807692 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3244031600 con 0x7f324c164800 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.933+0000 7f324a7fc640 1 -- 172.21.15.146:0/2450807692 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32440319d0 con 0x7f324c164800 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.933+0000 7f3251a92640 1 -- 172.21.15.146:0/2450807692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c164800 msgr2=0x7f324c160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.933+0000 7f3251a92640 1 --2- 172.21.15.146:0/2450807692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c164800 0x7f324c160830 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f324c0710e0 tx=0x7f324402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 -- 172.21.15.146:0/2450807692 shutdown_connections 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 --2- 172.21.15.146:0/2450807692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c164800 0x7f324c160830 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 --2- 172.21.15.146:0/2450807692 >> v2:172.21.15.177:3300/0 conn(0x7f324c163eb0 0x7f324c1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 --2- 172.21.15.146:0/2450807692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c06ee60 0x7f324c0b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 -- 172.21.15.146:0/2450807692 >> 172.21.15.146:0/2450807692 conn(0x7f324c0b52e0 msgr2=0x7f324c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:11.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 -- 172.21.15.146:0/2450807692 shutdown_connections 2024-04-25T15:22:11.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.934+0000 7f3251a92640 1 -- 172.21.15.146:0/2450807692 wait complete. 2024-04-25T15:22:11.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.935+0000 7f3251a92640 1 Processor -- start 2024-04-25T15:22:11.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.936+0000 7f3251a92640 1 -- start start 2024-04-25T15:22:11.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f3251a92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c06ee60 0x7f324c15d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f3251a92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 0x7f324c1ea150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f3251a92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c164800 0x7f324c1ea690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f3251a92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f324c08dbc0 con 0x7f324c163eb0 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f3251a92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f324c08d3c0 con 0x7f324c06ee60 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f3251a92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f324c164c10 con 0x7f324c164800 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 0x7f324c1ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 0x7f324c1ea150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57958/0 (socket says 172.21.15.146:57958) 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 -- 172.21.15.146:0/516502283 learned_addr learned my addr 172.21.15.146:0/516502283 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324b7fe640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c06ee60 0x7f324c15d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324bfff640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c164800 0x7f324c1ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 -- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c164800 msgr2=0x7f324c1ea690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c164800 0x7f324c1ea690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 -- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c06ee60 msgr2=0x7f324c15d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:11.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c06ee60 0x7f324c15d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 -- 172.21.15.146:0/516502283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c0b7d30 con 0x7f324c163eb0 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324b7fe640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c06ee60 0x7f324c15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.937+0000 7f324affd640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 0x7f324c1ea150 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f324000d620 tx=0x7f324000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.938+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324000b620 con 0x7f324c163eb0 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.938+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3240019070 con 0x7f324c163eb0 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.938+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324c1ec250 con 0x7f324c163eb0 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.938+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324000ba00 con 0x7f324c163eb0 2024-04-25T15:22:11.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.938+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324c1ec810 con 0x7f324c163eb0 2024-04-25T15:22:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.939+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f324c1f7660 con 0x7f324c163eb0 2024-04-25T15:22:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.939+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f324000bba0 con 0x7f324c163eb0 2024-04-25T15:22:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.940+0000 7f3233fff640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3238074570 0x7f3238076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.940+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f3240095480 con 0x7f324c163eb0 2024-04-25T15:22:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.940+0000 7f324b7fe640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3238074570 0x7f3238076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:11.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.940+0000 7f324b7fe640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3238074570 0x7f3238076a80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f323c00ba30 tx=0x7f323c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:11.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:11.944+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f324005fd40 con 0x7f324c163eb0 2024-04-25T15:22:12.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:12.187+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_dir_fragment"]} v 0) -- 0x7f324c06a0e0 con 0x7f324c163eb0 2024-04-25T15:22:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.187+0000 7f3233fff640 1 -- 172.21.15.146:0/516502283 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_dir_fragment"]}]=0 v159) ==== 145+0+0 (secure 0 0 0) 0x7f324005f640 con 0x7f324c163eb0 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3238074570 msgr2=0x7f3238076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3238074570 0x7f3238076a80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f323c00ba30 tx=0x7f323c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 msgr2=0x7f324c1ea150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 0x7f324c1ea150 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f324000d620 tx=0x7f324000dae0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 shutdown_connections 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3238074570 0x7f3238076a80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f324c164800 0x7f324c1ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c163eb0 0x7f324c1ea150 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 --2- 172.21.15.146:0/516502283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f324c06ee60 0x7f324c15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 >> 172.21.15.146:0/516502283 conn(0x7f324c0b52e0 msgr2=0x7f324c1632f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 shutdown_connections 2024-04-25T15:22:13.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.190+0000 7f3251a92640 1 -- 172.21.15.146:0/516502283 wait complete. 2024-04-25T15:22:13.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:22:13.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.326+0000 7f0ee9834640 1 Processor -- start 2024-04-25T15:22:13.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.326+0000 7f0ee9834640 1 -- start start 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee9834640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0ee408dd50 0x7f0ee40b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee9834640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0ee40b8f30 0x7f0ee40d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee9834640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee408d590 con 0x7f0ee40d2170 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee9834640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee40b9340 con 0x7f0ee40b8f30 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee9834640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee40cf700 con 0x7f0ee408dd50 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee27fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0ee40b8f30 0x7f0ee40d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0ee40d2170 0x7f0ee40b32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38112/0 (socket says 172.21.15.146:38112) 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee37fe640 1 -- 172.21.15.146:0/3638563237 learned_addr learned my addr 172.21.15.146:0/3638563237 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:13.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee2ffd640 1 --2- 172.21.15.146:0/3638563237 >> v2:172.21.15.177:3300/0 conn(0x7f0ee408dd50 0x7f0ee40b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee27fc640 1 -- 172.21.15.146:0/3638563237 >> v2:172.21.15.177:3300/0 conn(0x7f0ee408dd50 msgr2=0x7f0ee40b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee27fc640 1 --2- 172.21.15.146:0/3638563237 >> v2:172.21.15.177:3300/0 conn(0x7f0ee408dd50 0x7f0ee40b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee27fc640 1 -- 172.21.15.146:0/3638563237 >> v1:172.21.15.146:6789/0 conn(0x7f0ee40d2170 legacy=0x7f0ee40b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.327+0000 7f0ee27fc640 1 -- 172.21.15.146:0/3638563237 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee40d27d0 con 0x7f0ee40b8f30 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.328+0000 7f0ee27fc640 1 --2- 172.21.15.146:0/3638563237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40b8f30 0x7f0ee40d1c30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40b99b0 tx=0x7f0ed802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c9bcb2b1e238e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.328+0000 7f0ee1ffb640 1 -- 172.21.15.146:0/3638563237 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8037070 con 0x7f0ee40b8f30 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.328+0000 7f0ee1ffb640 1 -- 172.21.15.146:0/3638563237 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed8032440 con 0x7f0ee40b8f30 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.328+0000 7f0ee1ffb640 1 -- 172.21.15.146:0/3638563237 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8032810 con 0x7f0ee40b8f30 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 -- 172.21.15.146:0/3638563237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40b8f30 msgr2=0x7f0ee40d1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 --2- 172.21.15.146:0/3638563237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40b8f30 0x7f0ee40d1c30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40b99b0 tx=0x7f0ed802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 -- 172.21.15.146:0/3638563237 shutdown_connections 2024-04-25T15:22:13.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 --2- 172.21.15.146:0/3638563237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40b8f30 0x7f0ee40d1c30 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 --2- 172.21.15.146:0/3638563237 >> v2:172.21.15.177:3300/0 conn(0x7f0ee408dd50 0x7f0ee40b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 -- 172.21.15.146:0/3638563237 >> 172.21.15.146:0/3638563237 conn(0x7f0ee40b5430 msgr2=0x7f0ee40b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 -- 172.21.15.146:0/3638563237 shutdown_connections 2024-04-25T15:22:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.329+0000 7f0ee9834640 1 -- 172.21.15.146:0/3638563237 wait complete. 2024-04-25T15:22:13.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.331+0000 7f0ee9834640 1 Processor -- start 2024-04-25T15:22:13.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.331+0000 7f0ee9834640 1 -- start start 2024-04-25T15:22:13.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee9834640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ee40b8f30 0x7f0ee409e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee9834640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 0x7f0ee41f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee9834640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ee41f5430 0x7f0ee41e82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee9834640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee40d2580 con 0x7f0ee40d2170 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee9834640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee408d8c0 con 0x7f0ee40b8f30 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee9834640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee40d2ab0 con 0x7f0ee41f5430 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee27fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 0x7f0ee41f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee27fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 0x7f0ee41f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57986/0 (socket says 172.21.15.146:57986) 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.332+0000 7f0ee27fc640 1 -- 172.21.15.146:0/4242451595 learned_addr learned my addr 172.21.15.146:0/4242451595 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee2ffd640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ee40b8f30 0x7f0ee409e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee37fe640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ee41f5430 0x7f0ee41e82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee27fc640 1 -- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ee41f5430 msgr2=0x7f0ee41e82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee27fc640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ee41f5430 0x7f0ee41e82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee27fc640 1 -- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ee40b8f30 msgr2=0x7f0ee409e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee27fc640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ee40b8f30 0x7f0ee409e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee27fc640 1 -- 172.21.15.146:0/4242451595 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee409ee60 con 0x7f0ee40d2170 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee2ffd640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ee40b8f30 0x7f0ee409e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee27fc640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 0x7f0ee41f4ef0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8000c00 tx=0x7f0ed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8037040 con 0x7f0ee40d2170 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed8042070 con 0x7f0ee40d2170 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee409f140 con 0x7f0ee40d2170 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8030bd0 con 0x7f0ee40d2170 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.333+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee409f700 con 0x7f0ee40d2170 2024-04-25T15:22:13.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.334+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee409ebd0 con 0x7f0ee40d2170 2024-04-25T15:22:13.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.335+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f0ed8047460 con 0x7f0ee40d2170 2024-04-25T15:22:13.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.335+0000 7f0ecbfff640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ed0074570 0x7f0ed0076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.335+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f0ed80b5bd0 con 0x7f0ee40d2170 2024-04-25T15:22:13.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.335+0000 7f0ee2ffd640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ed0074570 0x7f0ed0076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.336+0000 7f0ee2ffd640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ed0074570 0x7f0ed0076a80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40d13f0 tx=0x7f0ed40039d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:13.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.340+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ed8032e20 con 0x7f0ee40d2170 2024-04-25T15:22:13.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.514+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f0ee4072a30 con 0x7f0ee40d2170 2024-04-25T15:22:13.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.518+0000 7f0ecbfff640 1 -- 172.21.15.146:0/4242451595 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f0ed8080410 con 0x7f0ee40d2170 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ed0074570 msgr2=0x7f0ed0076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ed0074570 0x7f0ed0076a80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40d13f0 tx=0x7f0ed40039d0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 msgr2=0x7f0ee41f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 0x7f0ee41f4ef0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8000c00 tx=0x7f0ed80046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 shutdown_connections 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0ed0074570 0x7f0ed0076a80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ee41f5430 0x7f0ee41e82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ee40d2170 0x7f0ee41f4ef0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 --2- 172.21.15.146:0/4242451595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ee40b8f30 0x7f0ee409e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 >> 172.21.15.146:0/4242451595 conn(0x7f0ee40b5430 msgr2=0x7f0ee40b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:13.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 shutdown_connections 2024-04-25T15:22:13.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.520+0000 7f0ee9834640 1 -- 172.21.15.146:0/4242451595 wait complete. 2024-04-25T15:22:13.532 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:22:13.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:22:13.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:22:13.664 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:22:13.664 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:13.664 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:13.682 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1449 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:22:13.682 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:13.682 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:13.722 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3659 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:22:13.722 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:13.722 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:13.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1449 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:22:13.738 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:13.738 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:13.779 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3659 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:22:13.779 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:22:13.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:13.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.886+0000 7f29f279f640 1 Processor -- start 2024-04-25T15:22:13.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.887+0000 7f29f279f640 1 -- start start 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f279f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f29ec0b49f0 0x7f29ec15fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f279f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec0d4e20 con 0x7f29ec0b8dd0 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f279f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec0d5100 con 0x7f29ec1640b0 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f279f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec0d4bd0 con 0x7f29ec0b49f0 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f179d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f29ec0b8dd0 0x7f29ec1639a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38124/0 (socket says 172.21.15.146:38124) 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f179d640 1 -- 172.21.15.146:0/1480656017 learned_addr learned my addr 172.21.15.146:0/1480656017 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:13.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229472548 0 0) 0x7f29ec0d4e20 con 0x7f29ec0b8dd0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29f0f9c640 1 --2- 172.21.15.146:0/1480656017 >> v2:172.21.15.177:3300/0 conn(0x7f29ec0b49f0 0x7f29ec15fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.888+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d8003520 con 0x7f29ec0b8dd0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3874188600 0 0) 0x7f29d8003520 con 0x7f29ec0b8dd0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 >> v2:172.21.15.177:3300/0 conn(0x7f29ec0b49f0 msgr2=0x7f29ec15fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 --2- 172.21.15.146:0/1480656017 >> v2:172.21.15.177:3300/0 conn(0x7f29ec0b49f0 0x7f29ec15fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 >> v1:172.21.15.177:6789/0 conn(0x7f29ec1640b0 legacy=0x7f29ec06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec06c7a0 con 0x7f29ec0b8dd0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f29e80019f0 con 0x7f29ec0b8dd0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 >> v1:172.21.15.146:6789/0 conn(0x7f29ec0b8dd0 legacy=0x7f29ec1639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29d80039b0 0x7f29d8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d8004800 0x7f29d8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29d8005270 0x7f29d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec0d4bd0 con 0x7f29d80039b0 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29d800c190 con 0x7f29d8004800 2024-04-25T15:22:13.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f29d800c390 con 0x7f29d8005270 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f179d640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29d80039b0 0x7f29d8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f0f9c640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d8004800 0x7f29d8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f1f9e640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29d8005270 0x7f29d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f179d640 1 -- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29d8005270 msgr2=0x7f29d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f179d640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29d8005270 0x7f29d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f179d640 1 -- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d8004800 msgr2=0x7f29d8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f179d640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d8004800 0x7f29d8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f179d640 1 -- 172.21.15.146:0/1480656017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d8003520 con 0x7f29d80039b0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.889+0000 7f29f1f9e640 1 -- 172.21.15.146:0/1480656017 reap_dead start 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.890+0000 7f29f179d640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29d80039b0 0x7f29d8003e40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f29e800a810 tx=0x7f29d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a6206b0c9edb23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.890+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e800a460 con 0x7f29d80039b0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.890+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e802f070 con 0x7f29d80039b0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.890+0000 7f29e3fff640 1 -- 172.21.15.146:0/1480656017 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e802bc90 con 0x7f29d80039b0 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 -- 172.21.15.146:0/1480656017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29d80039b0 msgr2=0x7f29d8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29d80039b0 0x7f29d8003e40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f29e800a810 tx=0x7f29d8005830 comp rx=0 tx=0).stop 2024-04-25T15:22:13.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 -- 172.21.15.146:0/1480656017 shutdown_connections 2024-04-25T15:22:13.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 --2- 172.21.15.146:0/1480656017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29d80039b0 0x7f29d8003e40 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 -- 172.21.15.146:0/1480656017 >> 172.21.15.146:0/1480656017 conn(0x7f29ec0d62f0 msgr2=0x7f29ec0d2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:13.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 -- 172.21.15.146:0/1480656017 shutdown_connections 2024-04-25T15:22:13.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.891+0000 7f29f279f640 1 -- 172.21.15.146:0/1480656017 wait complete. 2024-04-25T15:22:13.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.892+0000 7f29f279f640 1 Processor -- start 2024-04-25T15:22:13.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.892+0000 7f29f279f640 1 -- start start 2024-04-25T15:22:13.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f279f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d80039b0 0x7f29ec0cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f279f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 0x7f29ec0d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f279f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0d1030 0x7f29ec1eb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f279f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec06c550 con 0x7f29ec1fa070 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f279f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec1646d0 con 0x7f29d80039b0 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f279f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f29ec06cb10 con 0x7f29ec0d1030 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 0x7f29ec0d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 0x7f29ec0d0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58006/0 (socket says 172.21.15.146:58006) 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 -- 172.21.15.146:0/968987071 learned_addr learned my addr 172.21.15.146:0/968987071 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f179d640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d80039b0 0x7f29ec0cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f1f9e640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0d1030 0x7f29ec1eb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 -- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0d1030 msgr2=0x7f29ec1eb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0d1030 0x7f29ec1eb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 -- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d80039b0 msgr2=0x7f29ec0cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d80039b0 0x7f29ec0cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:13.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 -- 172.21.15.146:0/968987071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec0d1440 con 0x7f29ec1fa070 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29f0f9c640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 0x7f29ec0d0af0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f29dc011aa0 tx=0x7f29dc011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.894+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc00fb80 con 0x7f29ec1fa070 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.895+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29dc00fd20 con 0x7f29ec1fa070 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.895+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec160540 con 0x7f29ec1fa070 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.895+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc014660 con 0x7f29ec1fa070 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.895+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec1607e0 con 0x7f29ec1fa070 2024-04-25T15:22:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.896+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec163030 con 0x7f29ec1fa070 2024-04-25T15:22:13.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.897+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f29dc012040 con 0x7f29ec1fa070 2024-04-25T15:22:13.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.897+0000 7f29e1ffb640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f29d0074540 0x7f29d0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:13.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.897+0000 7f29f179d640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f29d0074540 0x7f29d0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:13.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.897+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f29dc095760 con 0x7f29ec1fa070 2024-04-25T15:22:13.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.898+0000 7f29f179d640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f29d0074540 0x7f29d0076a50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f29e8033000 tx=0x7f29e8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:13.902+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29ec0d1440 con 0x7f29ec1fa070 2024-04-25T15:22:14.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.116+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29ec050740 con 0x7f29ec1fa070 2024-04-25T15:22:14.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.117+0000 7f29e1ffb640 1 -- 172.21.15.146:0/968987071 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+3881 (secure 0 0 0) 0x7f29dc019030 con 0x7f29ec1fa070 2024-04-25T15:22:14.118 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:14.118 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3227137793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3227137793},{"type":"v1","addr":"172.21.15.146:6836","nonce":3227137793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5243,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1778965228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":1778965228},{"type":"v1","addr":"172.21.15.177:6835","nonce":1778965228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3712845234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3712845234},{"type":"v1","addr":"172.21.15.146:6837","nonce":3712845234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5866,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/1328556398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1328556398},{"type":"v1","addr":"172.21.15.177:6834","nonce":1328556398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[]} 2024-04-25T15:22:14.118 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f29d0074540 msgr2=0x7f29d0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f29d0074540 0x7f29d0076a50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f29e8033000 tx=0x7f29e8007000 comp rx=0 tx=0).stop 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 msgr2=0x7f29ec0d0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 0x7f29ec0d0af0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f29dc011aa0 tx=0x7f29dc011f60 comp rx=0 tx=0).stop 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 shutdown_connections 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f29d0074540 0x7f29d0076a50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0d1030 0x7f29ec1eb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29ec1fa070 0x7f29ec0d0af0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 --2- 172.21.15.146:0/968987071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29d80039b0 0x7f29ec0cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 >> 172.21.15.146:0/968987071 conn(0x7f29ec0d62f0 msgr2=0x7f29ec0b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:14.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 shutdown_connections 2024-04-25T15:22:14.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.119+0000 7f29f279f640 1 -- 172.21.15.146:0/968987071 wait complete. 2024-04-25T15:22:14.132 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:22:14.132 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:22:14.132 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:22:14.132 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:22:14.132 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:22:14.132 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:22:14.133 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:22:14.133 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:22:14.133 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:22:14.133 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:22:14.133 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:22:14.133 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:22:14.133 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:22:14.133 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:22:14.134 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:22:14.134 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:22:14.143 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:22:14.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:22:14.144 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:22:14.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:22:14.151 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:22:14.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:22:14.152 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:22:14.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:22:14.186 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:22:14.186 DEBUG:teuthology.parallel:result is None 2024-04-25T15:22:14.187 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:22:14.187 DEBUG:teuthology.parallel:result is None 2024-04-25T15:22:14.194 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:22:14.194 DEBUG:teuthology.parallel:result is None 2024-04-25T15:22:14.195 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:22:14.195 DEBUG:teuthology.parallel:result is None 2024-04-25T15:22:14.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:22:14.271 INFO:tasks.ceph.mds.a.smithi146.stdout:starting mds.a at 2024-04-25T15:22:14.271 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-04-25T15:22:14.272 INFO:tasks.ceph.mds.a.smithi146.stderr:2024-04-25T15:22:14.271+0000 7fa22f481640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-25T15:22:14.285 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-04-25T15:22:14.285 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-04-25T15:22:14.286 INFO:tasks.ceph.mds.d.smithi177.stderr:2024-04-25T15:22:14.285+0000 7f7647e5a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-25T15:22:14.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.310+0000 7ff4daa69640 1 Processor -- start 2024-04-25T15:22:14.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.310+0000 7ff4daa69640 1 -- start start 2024-04-25T15:22:14.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4daa69640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40d0e10 0x7ff4d415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:14.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4daa69640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff4d40b45d0 0x7ff4d406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4daa69640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d40d5000 con 0x7ff4d40d12a0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4daa69640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d40d4db0 con 0x7ff4d40b45d0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4daa69640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d40d52e0 con 0x7ff4d40d0e10 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4d8805640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff4d40d12a0 0x7ff4d40d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38148/0 (socket says 172.21.15.146:38148) 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4d8805640 1 -- 172.21.15.146:0/3328530682 learned_addr learned my addr 172.21.15.146:0/3328530682 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4d9006640 1 --2- 172.21.15.146:0/3328530682 >> v2:172.21.15.177:3300/0 conn(0x7ff4d40b45d0 0x7ff4d406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.311+0000 7ff4d3fff640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40d0e10 0x7ff4d415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d37fe640 1 -- 172.21.15.146:0/3328530682 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122836383 0 0) 0x7ff4d40d5000 con 0x7ff4d40d12a0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d37fe640 1 -- 172.21.15.146:0/3328530682 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4bc003520 con 0x7ff4d40d12a0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d9006640 1 -- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40d0e10 msgr2=0x7ff4d415ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d9006640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40d0e10 0x7ff4d415ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d9006640 1 -- 172.21.15.146:0/3328530682 >> v1:172.21.15.146:6789/0 conn(0x7ff4d40d12a0 legacy=0x7ff4d40d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d9006640 1 -- 172.21.15.146:0/3328530682 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d406c5f0 con 0x7ff4d40b45d0 2024-04-25T15:22:14.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d3fff640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40d0e10 0x7ff4d415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.312+0000 7ff4d9006640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40b45d0 0x7ff4d406cdf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009d90 tx=0x7ff4cc02a530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c202620ecce11b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4d37fe640 1 -- 172.21.15.146:0/3328530682 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc007740 con 0x7ff4d40b45d0 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4d37fe640 1 -- 172.21.15.146:0/3328530682 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4cc0078e0 con 0x7ff4d40b45d0 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4d37fe640 1 -- 172.21.15.146:0/3328530682 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc007c90 con 0x7ff4d40b45d0 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 -- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40b45d0 msgr2=0x7ff4d406cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40b45d0 0x7ff4d406cdf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009d90 tx=0x7ff4cc02a530 comp rx=0 tx=0).stop 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 -- 172.21.15.146:0/3328530682 shutdown_connections 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40b45d0 0x7ff4d406cdf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 --2- 172.21.15.146:0/3328530682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40d0e10 0x7ff4d415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 -- 172.21.15.146:0/3328530682 >> 172.21.15.146:0/3328530682 conn(0x7ff4d40d63b0 msgr2=0x7ff4d40d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:14.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 -- 172.21.15.146:0/3328530682 shutdown_connections 2024-04-25T15:22:14.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.313+0000 7ff4daa69640 1 -- 172.21.15.146:0/3328530682 wait complete. 2024-04-25T15:22:14.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.315+0000 7ff4daa69640 1 Processor -- start 2024-04-25T15:22:14.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.315+0000 7ff4daa69640 1 -- start start 2024-04-25T15:22:14.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40b45d0 0x7ff4d414cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:14.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 0x7ff4d414e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:14.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40ce170 0x7ff4d414ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d40b9090 con 0x7ff4d40d12a0 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d40b8f10 con 0x7ff4d40ce170 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d40b4b20 con 0x7ff4d40b45d0 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 0x7ff4d414e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 0x7ff4d414e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58046/0 (socket says 172.21.15.146:58046) 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 -- 172.21.15.146:0/2642143450 learned_addr learned my addr 172.21.15.146:0/2642143450 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d8805640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40b45d0 0x7ff4d414cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d9006640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40ce170 0x7ff4d414ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 -- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40b45d0 msgr2=0x7ff4d414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40b45d0 0x7ff4d414cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 -- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40ce170 msgr2=0x7ff4d414ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40ce170 0x7ff4d414ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:14.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 -- 172.21.15.146:0/2642143450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d414d800 con 0x7ff4d40d12a0 2024-04-25T15:22:14.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d3fff640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 0x7ff4d414e8a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800e990 tx=0x7ff4c800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:14.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8017070 con 0x7ff4d40d12a0 2024-04-25T15:22:14.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c800c9e0 con 0x7ff4d40d12a0 2024-04-25T15:22:14.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d414dae0 con 0x7ff4d40d12a0 2024-04-25T15:22:14.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800cdb0 con 0x7ff4d40d12a0 2024-04-25T15:22:14.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.317+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d414e0a0 con 0x7ff4d40d12a0 2024-04-25T15:22:14.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.318+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d406a0e0 con 0x7ff4d40d12a0 2024-04-25T15:22:14.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.319+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7ff4c801a070 con 0x7ff4d40d12a0 2024-04-25T15:22:14.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.319+0000 7ff4d17fa640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4c0074540 0x7ff4c0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:14.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.320+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7ff4c80953b0 con 0x7ff4d40d12a0 2024-04-25T15:22:14.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.320+0000 7ff4d8805640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4c0074540 0x7ff4c0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:14.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.320+0000 7ff4d8805640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4c0074540 0x7ff4c0076a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4000f60 tx=0x7ff4c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:14.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.324+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4c805fc70 con 0x7ff4d40d12a0 2024-04-25T15:22:14.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:14.513+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff4d415e4e0 con 0x7ff4d40d12a0 2024-04-25T15:22:15.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.529+0000 7ff4d17fa640 1 -- 172.21.15.146:0/2642143450 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v50) ==== 97+0+0 (secure 0 0 0) 0x7ff4c805f570 con 0x7ff4d40d12a0 2024-04-25T15:22:15.530 INFO:teuthology.orchestra.run.smithi146.stderr: removed all blocklist entries 2024-04-25T15:22:15.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4c0074540 msgr2=0x7ff4c0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4c0074540 0x7ff4c0076a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4000f60 tx=0x7ff4c4009040 comp rx=0 tx=0).stop 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 msgr2=0x7ff4d414e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 0x7ff4d414e8a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800e990 tx=0x7ff4c800ee50 comp rx=0 tx=0).stop 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 shutdown_connections 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff4c0074540 0x7ff4c0076a50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4d40ce170 0x7ff4d414ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4d40d12a0 0x7ff4d414e8a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 --2- 172.21.15.146:0/2642143450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4d40b45d0 0x7ff4d414cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 >> 172.21.15.146:0/2642143450 conn(0x7ff4d40d63b0 msgr2=0x7ff4d4160b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 shutdown_connections 2024-04-25T15:22:15.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.533+0000 7ff4daa69640 1 -- 172.21.15.146:0/2642143450 wait complete. 2024-04-25T15:22:15.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:22:15.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.660+0000 7fa92d09f640 1 Processor -- start 2024-04-25T15:22:15.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.660+0000 7fa92d09f640 1 -- start start 2024-04-25T15:22:15.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92d09f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa9280cc280 0x7fa928070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:15.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92d09f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa92806c8c0 0x7fa92806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92d09f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9280cd490 con 0x7fa9281618f0 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92d09f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa92815bcd0 con 0x7fa92806c8c0 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92d09f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa9280711d0 con 0x7fa9280cc280 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92759e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa92806c8c0 0x7fa92806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92759e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa92806c8c0 0x7fa92806ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58058/0 (socket says 172.21.15.146:58058) 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.661+0000 7fa92759e640 1 -- 172.21.15.146:0/162442992 learned_addr learned my addr 172.21.15.146:0/162442992 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa926d9d640 1 --2- 172.21.15.146:0/162442992 >> v2:172.21.15.177:3300/0 conn(0x7fa9280cc280 0x7fa928070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa92759e640 1 -- 172.21.15.146:0/162442992 >> v2:172.21.15.177:3300/0 conn(0x7fa9280cc280 msgr2=0x7fa928070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa92759e640 1 --2- 172.21.15.146:0/162442992 >> v2:172.21.15.177:3300/0 conn(0x7fa9280cc280 0x7fa928070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa92759e640 1 -- 172.21.15.146:0/162442992 >> v1:172.21.15.177:6789/0 conn(0x7fa9281618f0 legacy=0x7fa928161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa92759e640 1 -- 172.21.15.146:0/162442992 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280cc880 con 0x7fa92806c8c0 2024-04-25T15:22:15.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa926d9d640 1 --2- 172.21.15.146:0/162442992 >> v2:172.21.15.177:3300/0 conn(0x7fa9280cc280 0x7fa928070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.662+0000 7fa92759e640 1 --2- 172.21.15.146:0/162442992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa92806c8c0 0x7fa92806ccd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa928071700 tx=0x7fa92002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd663ed6d1b9a2bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92659c640 1 -- 172.21.15.146:0/162442992 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920036070 con 0x7fa92806c8c0 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92659c640 1 -- 172.21.15.146:0/162442992 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa920031600 con 0x7fa92806c8c0 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92659c640 1 -- 172.21.15.146:0/162442992 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9200319d0 con 0x7fa92806c8c0 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 -- 172.21.15.146:0/162442992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa92806c8c0 msgr2=0x7fa92806ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 --2- 172.21.15.146:0/162442992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa92806c8c0 0x7fa92806ccd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa928071700 tx=0x7fa92002a610 comp rx=0 tx=0).stop 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 -- 172.21.15.146:0/162442992 shutdown_connections 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 --2- 172.21.15.146:0/162442992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa92806c8c0 0x7fa92806ccd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 --2- 172.21.15.146:0/162442992 >> v2:172.21.15.177:3300/0 conn(0x7fa9280cc280 0x7fa928070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 -- 172.21.15.146:0/162442992 >> 172.21.15.146:0/162442992 conn(0x7fa9280ce0c0 msgr2=0x7fa9280ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 -- 172.21.15.146:0/162442992 shutdown_connections 2024-04-25T15:22:15.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.663+0000 7fa92d09f640 1 -- 172.21.15.146:0/162442992 wait complete. 2024-04-25T15:22:15.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.665+0000 7fa92d09f640 1 Processor -- start 2024-04-25T15:22:15.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.665+0000 7fa92d09f640 1 -- start start 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92806c8c0 0x7fa9280b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 0x7fa92814bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9281618f0 0x7fa9280b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280cd120 con 0x7fa9280cc280 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280cce40 con 0x7fa92806c8c0 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280b0940 con 0x7fa9281618f0 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 0x7fa92814bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 0x7fa92814bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58072/0 (socket says 172.21.15.146:58072) 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 -- 172.21.15.146:0/3763319037 learned_addr learned my addr 172.21.15.146:0/3763319037 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa926d9d640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92806c8c0 0x7fa9280b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:15.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92759e640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9281618f0 0x7fa9280b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 -- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9281618f0 msgr2=0x7fa9280b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9281618f0 0x7fa9280b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 -- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92806c8c0 msgr2=0x7fa9280b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92806c8c0 0x7fa9280b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 -- 172.21.15.146:0/3763319037 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280b5330 con 0x7fa9280cc280 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92759e640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9281618f0 0x7fa9280b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa926d9d640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92806c8c0 0x7fa9280b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91ffff640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 0x7fa92814bed0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c3f0 tx=0x7fa91000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9100052d0 con 0x7fa9280cc280 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa910005490 con 0x7fa9280cc280 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9280b4d70 con 0x7fa9280cc280 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910007260 con 0x7fa9280cc280 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.667+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9281e77b0 con 0x7fa9280cc280 2024-04-25T15:22:15.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.668+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa92806a0e0 con 0x7fa9280cc280 2024-04-25T15:22:15.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.669+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa910007400 con 0x7fa9280cc280 2024-04-25T15:22:15.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.669+0000 7fa91f7fe640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa908074540 0x7fa908076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:15.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.670+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5249+0+0 (secure 0 0 0) 0x7fa910094530 con 0x7fa9280cc280 2024-04-25T15:22:15.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.670+0000 7fa926d9d640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa908074540 0x7fa908076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:15.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.670+0000 7fa926d9d640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa908074540 0x7fa908076a50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa918006fd0 tx=0x7fa918008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:15.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.674+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa91005ee90 con 0x7fa9280cc280 2024-04-25T15:22:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.914+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa92806fbb0 con 0x7fa9280cc280 2024-04-25T15:22:15.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.914+0000 7fa91f7fe640 1 -- 172.21.15.146:0/3763319037 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7fa91005e790 con 0x7fa9280cc280 2024-04-25T15:22:15.915 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:15.915 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:15.915 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_dump": [ 2024-04-25T15:22:15.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.915 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.a", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpm/Ux1DhAA3DlkdKegZpZxlaZPFjodBw==", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.b", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmR1PoGhAAw/gk2lg0yvzUcOUZvwo6jg==", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.916 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.c", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmtDPsFRAAh1HLdvyYLR78fokIKaekrA==", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mds.d", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpm3EVMIhAAddbxnaiTto+gXb0+LGbRKA==", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.0", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBcdCpmkFKVABAA+xGQMOacMJi18fh2p3rJFA==", 2024-04-25T15:22:15.918 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.1", 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBedCpm78w0OxAAGf4e1jZ0aAxnX/HBagE3yg==", 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.2", 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBhdCpm2VCWORAAUEOB2XJBAbm1STvjkbjFNw==", 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.3", 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBkdCpmfHmAOBAApJr+ddhMkB3As5PU2KALaQ==", 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.920 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.4", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBqdCpmtoKbORAA63fHSceN+QJCRCuFLZKlqA==", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.5", 2024-04-25T15:22:15.921 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBtdCpmIkKAOBAAN4te1g8Ioao/xQtjvw4vSg==", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.6", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBwdCpm90+YNhAApJ3X6R6BJC2VXiJ+RCl+TA==", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.922 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "osd.7", 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBzdCpmACtMNxAAFvU4+bypsr14tXk8suRhEQ==", 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow profile osd", 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile osd", 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.0", 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmlczBJhAApZj6jMADYxBHeBvhaprLDg==", 2024-04-25T15:22:15.923 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow r", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.1", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmy/P3LRAAyZkQ/mpIZ9/H8m7/scVDKA==", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:22:15.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.2", 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmhV1cKhAAdJQ+5g5RjIlilhKuxLcyiA==", 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.3", 2024-04-25T15:22:15.925 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmR6yLMRAAhFRT/TgCAnREAuUr53J1BA==", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow r", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow rw", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow rwx" 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "client.admin", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBXdCpmiIoCLRAAnMVBi4QtXLwcDZ/1klNNcg==", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow *", 2024-04-25T15:22:15.926 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": "allow *", 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow *", 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mgr.x", 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmiMrSChAA/AOecro7QL+ypBbUDiwShQ==", 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow *", 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile mgr", 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:15.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": "mgr.y", 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "AQBYdCpmmk8XBhAASX5Dc72zChEYixIh/iiawg==", 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": { 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": "allow *", 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: "mon": "allow profile mgr", 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": "allow *" 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-25T15:22:15.928 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa908074540 msgr2=0x7fa908076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa908074540 0x7fa908076a50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa918006fd0 tx=0x7fa918008040 comp rx=0 tx=0).stop 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 msgr2=0x7fa92814bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 0x7fa92814bed0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c3f0 tx=0x7fa91000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 shutdown_connections 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa908074540 0x7fa908076a50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9281618f0 0x7fa9280b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9280cc280 0x7fa92814bed0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 --2- 172.21.15.146:0/3763319037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa92806c8c0 0x7fa9280b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 >> 172.21.15.146:0/3763319037 conn(0x7fa9280ce0c0 msgr2=0x7fa92815aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 shutdown_connections 2024-04-25T15:22:15.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:15.916+0000 7fa92d09f640 1 -- 172.21.15.146:0/3763319037 wait complete. 2024-04-25T15:22:15.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:16.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.039+0000 7fb750651640 1 Processor -- start 2024-04-25T15:22:16.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.039+0000 7fb750651640 1 -- start start 2024-04-25T15:22:16.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb750651640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb7480d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb750651640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb74806ccc0 0x7fb74806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb750651640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7480b9200 con 0x7fb7480d0020 2024-04-25T15:22:16.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb750651640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb74806c550 con 0x7fb74806ccc0 2024-04-25T15:22:16.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb750651640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb74806c830 con 0x7fb7480d4f60 2024-04-25T15:22:16.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb74ebee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb74806ccc0 0x7fb74806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb74ebee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb74806ccc0 0x7fb74806d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58082/0 (socket says 172.21.15.146:58082) 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb74dbec640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb7480d0020 0x7fb7480d0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38158/0 (socket says 172.21.15.146:38158) 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb74ebee640 1 -- 172.21.15.146:0/1278149125 learned_addr learned my addr 172.21.15.146:0/1278149125 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.040+0000 7fb74e3ed640 1 --2- 172.21.15.146:0/1278149125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb7480d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.041+0000 7fb74ebee640 1 -- 172.21.15.146:0/1278149125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 msgr2=0x7fb7480d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.041+0000 7fb74ebee640 1 --2- 172.21.15.146:0/1278149125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb7480d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.041+0000 7fb74ebee640 1 -- 172.21.15.146:0/1278149125 >> v1:172.21.15.146:6789/0 conn(0x7fb7480d0020 legacy=0x7fb7480d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.041+0000 7fb74ebee640 1 -- 172.21.15.146:0/1278149125 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7480d1070 con 0x7fb74806ccc0 2024-04-25T15:22:16.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.041+0000 7fb74ebee640 1 --2- 172.21.15.146:0/1278149125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb74806ccc0 0x7fb74806d0d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb740009870 tx=0x7fb74002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e030f87ddd53c4e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.041+0000 7fb74d3eb640 1 -- 172.21.15.146:0/1278149125 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb740037070 con 0x7fb74806ccc0 2024-04-25T15:22:16.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb74d3eb640 1 -- 172.21.15.146:0/1278149125 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb740032440 con 0x7fb74806ccc0 2024-04-25T15:22:16.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb74d3eb640 1 -- 172.21.15.146:0/1278149125 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb740032810 con 0x7fb74806ccc0 2024-04-25T15:22:16.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 -- 172.21.15.146:0/1278149125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb74806ccc0 msgr2=0x7fb74806d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 --2- 172.21.15.146:0/1278149125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb74806ccc0 0x7fb74806d0d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb740009870 tx=0x7fb74002a130 comp rx=0 tx=0).stop 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 -- 172.21.15.146:0/1278149125 shutdown_connections 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 --2- 172.21.15.146:0/1278149125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb74806ccc0 0x7fb74806d0d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 --2- 172.21.15.146:0/1278149125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb7480d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 -- 172.21.15.146:0/1278149125 >> 172.21.15.146:0/1278149125 conn(0x7fb7480d2450 msgr2=0x7fb7480d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 -- 172.21.15.146:0/1278149125 shutdown_connections 2024-04-25T15:22:16.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.042+0000 7fb750651640 1 -- 172.21.15.146:0/1278149125 wait complete. 2024-04-25T15:22:16.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.044+0000 7fb750651640 1 Processor -- start 2024-04-25T15:22:16.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.044+0000 7fb750651640 1 -- start start 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.045+0000 7fb750651640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb74806ccc0 0x7fb748161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.045+0000 7fb750651640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 0x7fb7481f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.045+0000 7fb750651640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb74815e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.045+0000 7fb750651640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb7480b5150 con 0x7fb7480d0020 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.045+0000 7fb750651640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7480d15f0 con 0x7fb74806ccc0 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.045+0000 7fb750651640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb7480b52d0 con 0x7fb7480d4f60 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 0x7fb7481f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 0x7fb7481f5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58098/0 (socket says 172.21.15.146:58098) 2024-04-25T15:22:16.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 -- 172.21.15.146:0/2146107081 learned_addr learned my addr 172.21.15.146:0/2146107081 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74e3ed640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb74806ccc0 0x7fb748161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74ebee640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb74815e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 -- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 msgr2=0x7fb74815e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb74815e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 -- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb74806ccc0 msgr2=0x7fb748161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb74806ccc0 0x7fb748161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 -- 172.21.15.146:0/2146107081 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7481f63c0 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb74dbec640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 0x7fb7481f5e80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb73800c390 tx=0x7fb73800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7380074e0 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7380047e0 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb748161900 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb738004bd0 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.046+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb748161ec0 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.047+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7480cfe20 con 0x7fb7480d0020 2024-04-25T15:22:16.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.048+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fb73801a020 con 0x7fb7480d0020 2024-04-25T15:22:16.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.049+0000 7fb73effd640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb72c074540 0x7fb72c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.049+0000 7fb74e3ed640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb72c074540 0x7fb72c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.049+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5249+0+0 (secure 0 0 0) 0x7fb7380946d0 con 0x7fb7480d0020 2024-04-25T15:22:16.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.049+0000 7fb74e3ed640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb72c074540 0x7fb72c076a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7480b4880 tx=0x7fb74400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.053+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7480cfe20 con 0x7fb7480d0020 2024-04-25T15:22:16.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.229+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb74806a0e0 con 0x7fb7480d0020 2024-04-25T15:22:16.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.230+0000 7fb73effd640 1 -- 172.21.15.146:0/2146107081 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+13212 (secure 0 0 0) 0x7fb7480b9200 con 0x7fb7480d0020 2024-04-25T15:22:16.230 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:16.231 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":50,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:15.528274+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb72c074540 msgr2=0x7fb72c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb72c074540 0x7fb72c076a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7480b4880 tx=0x7fb74400b040 comp rx=0 tx=0).stop 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 msgr2=0x7fb7481f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 0x7fb7481f5e80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb73800c390 tx=0x7fb73800c790 comp rx=0 tx=0).stop 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 shutdown_connections 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb72c074540 0x7fb72c076a50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7480d4f60 0x7fb74815e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7480d0020 0x7fb7481f5e80 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 --2- 172.21.15.146:0/2146107081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb74806ccc0 0x7fb748161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 >> 172.21.15.146:0/2146107081 conn(0x7fb7480d2450 msgr2=0x7fb7480b5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 shutdown_connections 2024-04-25T15:22:16.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.232+0000 7fb750651640 1 -- 172.21.15.146:0/2146107081 wait complete. 2024-04-25T15:22:16.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:16.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:16.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:22:16.290 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:22:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.356+0000 7f1271205640 1 Processor -- start 2024-04-25T15:22:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.356+0000 7f1271205640 1 -- start start 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1271205640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 0x7f126c0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1271205640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f126c0d3300 0x7f126c071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1271205640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f126c071f70 con 0x7f126c071b60 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1271205640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f126c0d2260 con 0x7f126c0d3300 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1271205640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f126c0d2540 con 0x7f126c0b2cd0 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f126b59e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f126c071b60 0x7f126c08cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38172/0 (socket says 172.21.15.146:38172) 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f126b59e640 1 -- 172.21.15.146:0/3742457762 learned_addr learned my addr 172.21.15.146:0/3742457762 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f126ad9d640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 0x7f126c0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f126a59c640 1 --2- 172.21.15.146:0/3742457762 >> v2:172.21.15.177:3300/0 conn(0x7f126c0d3300 0x7f126c071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053734030 0 0) 0x7f1264001490 con 0x7f126c071b60 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1254003520 con 0x7f126c071b60 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1437913111 0 0) 0x7f1254003520 con 0x7f126c071b60 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 msgr2=0x7f126c0d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 0x7f126c0d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 >> v2:172.21.15.177:3300/0 conn(0x7f126c0d3300 msgr2=0x7f126c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 --2- 172.21.15.146:0/3742457762 >> v2:172.21.15.177:3300/0 conn(0x7f126c0d3300 0x7f126c071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c0d2820 con 0x7f126c071b60 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f1264001bc0 con 0x7f126c071b60 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 >> v1:172.21.15.146:6789/0 conn(0x7f126c071b60 legacy=0x7f126c08cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 0x7f12540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1254003f30 0x7f1254004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.358+0000 7f1269d9b640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254004d40 0x7f1254005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0d2260 con 0x7f1254003f30 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0d2540 con 0x7f1254004d40 2024-04-25T15:22:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f125400bac0 con 0x7f126c0b2cd0 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126a59c640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1254003f30 0x7f1254004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126ad9d640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 0x7f12540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126b59e640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254004d40 0x7f1254005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126a59c640 1 -- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 msgr2=0x7f12540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126a59c640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0b2cd0 0x7f12540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126a59c640 1 -- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254004d40 msgr2=0x7f1254005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126a59c640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254004d40 0x7f1254005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.359+0000 7f126a59c640 1 -- 172.21.15.146:0/3742457762 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1254003520 con 0x7f1254003f30 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f126a59c640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1254003f30 0x7f1254004800 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f126c075280 tx=0x7f1254005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56c57cb4241ff25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260036070 con 0x7f1254003f30 2024-04-25T15:22:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1260031400 con 0x7f1254003f30 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1269d9b640 1 -- 172.21.15.146:0/3742457762 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260009690 con 0x7f1254003f30 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1271205640 1 -- 172.21.15.146:0/3742457762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1254003f30 msgr2=0x7f1254004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1271205640 1 --2- 172.21.15.146:0/3742457762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1254003f30 0x7f1254004800 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f126c075280 tx=0x7f1254005830 comp rx=0 tx=0).stop 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f126b59e640 1 -- 172.21.15.146:0/3742457762 reap_dead start 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1271205640 1 -- 172.21.15.146:0/3742457762 shutdown_connections 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1271205640 1 -- 172.21.15.146:0/3742457762 >> 172.21.15.146:0/3742457762 conn(0x7f126c0b52e0 msgr2=0x7f126c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1271205640 1 -- 172.21.15.146:0/3742457762 shutdown_connections 2024-04-25T15:22:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.360+0000 7f1271205640 1 -- 172.21.15.146:0/3742457762 wait complete. 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.362+0000 7f1271205640 1 Processor -- start 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.362+0000 7f1271205640 1 -- start start 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254003f30 0x7f126c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0ca8a0 0x7f126c0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 0x7f126c1f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0cfb70 con 0x7f126c0c8390 2024-04-25T15:22:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0d2de0 con 0x7f1254003f30 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f126c08dbf0 con 0x7f126c0ca8a0 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 0x7f126c1f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 0x7f126c1f61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58108/0 (socket says 172.21.15.146:58108) 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 -- 172.21.15.146:0/283532025 learned_addr learned my addr 172.21.15.146:0/283532025 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126a59c640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0ca8a0 0x7f126c0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126ad9d640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254003f30 0x7f126c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 -- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0ca8a0 msgr2=0x7f126c0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0ca8a0 0x7f126c0c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 -- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254003f30 msgr2=0x7f126c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254003f30 0x7f126c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 -- 172.21.15.146:0/283532025 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c1f6f60 con 0x7f126c0c8390 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126a59c640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0ca8a0 0x7f126c0c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126ad9d640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254003f30 0x7f126c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f126b59e640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 0x7f126c1f61d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f126c071070 tx=0x7f1264004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640078f0 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1264018070 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126c0c7370 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264013630 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.364+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f126c0cacb0 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.366+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f126c1f6cb0 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.366+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f12640137d0 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.367+0000 7f12537fe640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1258074570 0x7f1258076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.367+0000 7f126ad9d640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1258074570 0x7f1258076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.367+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5249+0+0 (secure 0 0 0) 0x7f1264095340 con 0x7f126c0c8390 2024-04-25T15:22:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.367+0000 7f126ad9d640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1258074570 0x7f1258076a80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f125c006fd0 tx=0x7f125c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.371+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f126c071f70 con 0x7f126c0c8390 2024-04-25T15:22:16.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.545+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f126c06a0e0 con 0x7f126c0c8390 2024-04-25T15:22:16.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.546+0000 7f12537fe640 1 -- 172.21.15.146:0/283532025 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) ==== 93+0+10 (secure 0 0 0) 0x7f126405fd30 con 0x7f126c0c8390 2024-04-25T15:22:16.547 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:16.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1258074570 msgr2=0x7f1258076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1258074570 0x7f1258076a80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f125c006fd0 tx=0x7f125c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:16.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 msgr2=0x7f126c1f61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 0x7f126c1f61d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f126c071070 tx=0x7f1264004f20 comp rx=0 tx=0).stop 2024-04-25T15:22:16.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 shutdown_connections 2024-04-25T15:22:16.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1258074570 0x7f1258076a80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f126c0c8390 0x7f126c1f61d0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f126c0ca8a0 0x7f126c0c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 --2- 172.21.15.146:0/283532025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1254003f30 0x7f126c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 >> 172.21.15.146:0/283532025 conn(0x7f126c0b52e0 msgr2=0x7f126c0cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 shutdown_connections 2024-04-25T15:22:16.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.548+0000 7f1271205640 1 -- 172.21.15.146:0/283532025 wait complete. 2024-04-25T15:22:16.560 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:22:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.708+0000 7f7647c67640 1 Processor -- start 2024-04-25T15:22:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.708+0000 7f7647c67640 1 -- start start 2024-04-25T15:22:16.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.709+0000 7f7647c67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 0x7f76400cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.709+0000 7f7647c67640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f76400d19d0 0x7f76400d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.709+0000 7f7647c67640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76400b4a60 con 0x7f76400b3920 2024-04-25T15:22:16.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.709+0000 7f7647c67640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f76400b5020 con 0x7f76400d19d0 2024-04-25T15:22:16.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.709+0000 7f7647c67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f76400b4d40 con 0x7f76400cfbe0 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 0x7f76400cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 0x7f76400cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:42720/0 (socket says 172.21.15.146:42720) 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 -- 172.21.15.146:0/4076698401 learned_addr learned my addr 172.21.15.146:0/4076698401 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645202640 1 --2- 172.21.15.146:0/4076698401 >> v2:172.21.15.177:3300/0 conn(0x7f76400d19d0 0x7f76400d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 -- 172.21.15.146:0/4076698401 >> v2:172.21.15.177:3300/0 conn(0x7f76400d19d0 msgr2=0x7f76400d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 --2- 172.21.15.146:0/4076698401 >> v2:172.21.15.177:3300/0 conn(0x7f76400d19d0 0x7f76400d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 -- 172.21.15.146:0/4076698401 >> v1:172.21.15.177:6789/0 conn(0x7f76400b3920 legacy=0x7f76400b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7645a03640 1 -- 172.21.15.146:0/4076698401 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76400cead0 con 0x7f76400cfbe0 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.710+0000 7f7644a01640 1 -- 172.21.15.146:0/4076698401 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3820402950 0 0) 0x7f76400b4a60 con 0x7f76400b3920 2024-04-25T15:22:16.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.711+0000 7f7645a03640 1 --2- 172.21.15.146:0/4076698401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 0x7f76400cf260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f763c009850 tx=0x7f763c02a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f437cf2729c53a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.711+0000 7f7644a01640 1 -- 172.21.15.146:0/4076698401 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763c004230 con 0x7f76400cfbe0 2024-04-25T15:22:16.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.711+0000 7f7644a01640 1 -- 172.21.15.146:0/4076698401 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f763c0043d0 con 0x7f76400cfbe0 2024-04-25T15:22:16.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.711+0000 7f7644a01640 1 -- 172.21.15.146:0/4076698401 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763c003810 con 0x7f76400cfbe0 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 -- 172.21.15.146:0/4076698401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 msgr2=0x7f76400cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 --2- 172.21.15.146:0/4076698401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 0x7f76400cf260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f763c009850 tx=0x7f763c02a2c0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 -- 172.21.15.146:0/4076698401 shutdown_connections 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 --2- 172.21.15.146:0/4076698401 >> v2:172.21.15.177:3300/0 conn(0x7f76400d19d0 0x7f76400d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 --2- 172.21.15.146:0/4076698401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400cfbe0 0x7f76400cf260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 -- 172.21.15.146:0/4076698401 >> 172.21.15.146:0/4076698401 conn(0x7f76400b73b0 msgr2=0x7f76400b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:16.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 -- 172.21.15.146:0/4076698401 shutdown_connections 2024-04-25T15:22:16.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.712+0000 7f7647c67640 1 -- 172.21.15.146:0/4076698401 wait complete. 2024-04-25T15:22:16.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.714+0000 7f7647c67640 1 Processor -- start 2024-04-25T15:22:16.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.714+0000 7f7647c67640 1 -- start start 2024-04-25T15:22:16.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7647c67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400b3920 0x7f76400c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7647c67640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76400cfbe0 0x7f764014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7647c67640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 0x7f764014ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7647c67640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f76400cf7a0 con 0x7f76400d19d0 2024-04-25T15:22:16.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7647c67640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76400d2320 con 0x7f76400cfbe0 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7647c67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f764014d730 con 0x7f76400b3920 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7646204640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 0x7f764014ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7646204640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 0x7f764014ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58122/0 (socket says 172.21.15.146:58122) 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.715+0000 7f7646204640 1 -- 172.21.15.146:0/1625989757 learned_addr learned my addr 172.21.15.146:0/1625989757 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7645a03640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400b3920 0x7f76400c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7645202640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76400cfbe0 0x7f764014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7646204640 1 -- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400b3920 msgr2=0x7f76400c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7646204640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400b3920 0x7f76400c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7646204640 1 -- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76400cfbe0 msgr2=0x7f764014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7646204640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76400cfbe0 0x7f764014c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7646204640 1 -- 172.21.15.146:0/1625989757 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764014d230 con 0x7f76400d19d0 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7646204640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 0x7f764014ccf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f763800c710 tx=0x7f763800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638017070 con 0x7f76400d19d0 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7638003da0 con 0x7f76400d19d0 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638005290 con 0x7f76400d19d0 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76401e8d40 con 0x7f76400d19d0 2024-04-25T15:22:16.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.716+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76401e9300 con 0x7f76400d19d0 2024-04-25T15:22:16.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.717+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f764006a0e0 con 0x7f76400d19d0 2024-04-25T15:22:16.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.718+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f763801a070 con 0x7f76400d19d0 2024-04-25T15:22:16.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.718+0000 7f76367fc640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7624074540 0x7f7624076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:16.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.718+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5249+0+0 (secure 0 0 0) 0x7f76380945a0 con 0x7f76400d19d0 2024-04-25T15:22:16.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.718+0000 7f7645a03640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7624074540 0x7f7624076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:16.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.719+0000 7f7645a03640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7624074540 0x7f7624076a50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f763c033000 tx=0x7f763c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:16.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.723+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f763800ce20 con 0x7f76400d19d0 2024-04-25T15:22:16.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.899+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f76400cf920 con 0x7f76400d19d0 2024-04-25T15:22:16.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.899+0000 7f76367fc640 1 -- 172.21.15.146:0/1625989757 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) ==== 84+0+65 (secure 0 0 0) 0x7f763805e950 con 0x7f76400d19d0 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout: "poolnum": 1, 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout: "poolname": ".mgr" 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:16.901 INFO:teuthology.orchestra.run.smithi146.stdout:] 2024-04-25T15:22:16.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7624074540 msgr2=0x7f7624076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7624074540 0x7f7624076a50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f763c033000 tx=0x7f763c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:16.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 msgr2=0x7f764014ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 0x7f764014ccf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f763800c710 tx=0x7f763800cbd0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 shutdown_connections 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7624074540 0x7f7624076a50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76400d19d0 0x7f764014ccf0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76400cfbe0 0x7f764014c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 --2- 172.21.15.146:0/1625989757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76400b3920 0x7f76400c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 >> 172.21.15.146:0/1625989757 conn(0x7f76400b73b0 msgr2=0x7f764015fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 shutdown_connections 2024-04-25T15:22:16.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:16.901+0000 7f7647c67640 1 -- 172.21.15.146:0/1625989757 wait complete. 2024-04-25T15:22:16.913 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:22:16.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:22:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.025+0000 7f2c6db9e640 1 Processor -- start 2024-04-25T15:22:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.025+0000 7f2c6db9e640 1 -- start start 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c6db9e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c6815ffc0 0x7f2c6806e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c6db9e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c680d17f0 0x7f2c680d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c6db9e640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2c680d26b0 0x7f2c680d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c6db9e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f2c6806d7a0 con 0x7f2c680d17f0 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c6db9e640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2c680d3000 con 0x7f2c680d26b0 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c6db9e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2c680d2350 con 0x7f2c6815ffc0 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c680d17f0 0x7f2c680d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c680d17f0 0x7f2c680d1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58126/0 (socket says 172.21.15.146:58126) 2024-04-25T15:22:17.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 -- 172.21.15.146:0/1270640592 learned_addr learned my addr 172.21.15.146:0/1270640592 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c67fff640 1 --2- 172.21.15.146:0/1270640592 >> v2:172.21.15.177:3300/0 conn(0x7f2c680d26b0 0x7f2c680d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c677fe640 1 --2- 172.21.15.146:0/1270640592 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c6815ffc0 0x7f2c6806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 -- 172.21.15.146:0/1270640592 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c6815ffc0 msgr2=0x7f2c6806e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/1270640592 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c6815ffc0 0x7f2c6806e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 -- 172.21.15.146:0/1270640592 >> v2:172.21.15.177:3300/0 conn(0x7f2c680d26b0 msgr2=0x7f2c680d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/1270640592 >> v2:172.21.15.177:3300/0 conn(0x7f2c680d26b0 0x7f2c680d2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.027+0000 7f2c66ffd640 1 -- 172.21.15.146:0/1270640592 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6806da40 con 0x7f2c680d17f0 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.028+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/1270640592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c680d1c00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2c680d1260 tx=0x7f2c5c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7587253e21213eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.028+0000 7f2c667fc640 1 -- 172.21.15.146:0/1270640592 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c037070 con 0x7f2c680d17f0 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.028+0000 7f2c667fc640 1 -- 172.21.15.146:0/1270640592 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5c032440 con 0x7f2c680d17f0 2024-04-25T15:22:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.028+0000 7f2c667fc640 1 -- 172.21.15.146:0/1270640592 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c032810 con 0x7f2c680d17f0 2024-04-25T15:22:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 -- 172.21.15.146:0/1270640592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 msgr2=0x7f2c680d1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/1270640592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c680d1c00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2c680d1260 tx=0x7f2c5c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 -- 172.21.15.146:0/1270640592 shutdown_connections 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/1270640592 >> v2:172.21.15.177:3300/0 conn(0x7f2c680d26b0 0x7f2c680d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/1270640592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c680d1c00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/1270640592 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c6815ffc0 0x7f2c6806e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 -- 172.21.15.146:0/1270640592 >> 172.21.15.146:0/1270640592 conn(0x7f2c680b7460 msgr2=0x7f2c680b7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 -- 172.21.15.146:0/1270640592 shutdown_connections 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.029+0000 7f2c6db9e640 1 -- 172.21.15.146:0/1270640592 wait complete. 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.031+0000 7f2c6db9e640 1 Processor -- start 2024-04-25T15:22:17.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.031+0000 7f2c6db9e640 1 -- start start 2024-04-25T15:22:17.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c6db9e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c681f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c6db9e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c680d26b0 0x7f2c680ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c6db9e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6815ffc0 0x7f2c6815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c6db9e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2c681603d0 con 0x7f2c680d17f0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c6db9e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6806e000 con 0x7f2c6815ffc0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c6db9e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6806f760 con 0x7f2c680d26b0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c677fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c681f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c677fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c681f58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58134/0 (socket says 172.21.15.146:58134) 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c677fe640 1 -- 172.21.15.146:0/3008704300 learned_addr learned my addr 172.21.15.146:0/3008704300 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c680d26b0 0x7f2c680ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c67fff640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6815ffc0 0x7f2c6815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.032+0000 7f2c677fe640 1 -- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c680d26b0 msgr2=0x7f2c680ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c677fe640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c680d26b0 0x7f2c680ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c677fe640 1 -- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6815ffc0 msgr2=0x7f2c6815b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c677fe640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6815ffc0 0x7f2c6815b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c677fe640 1 -- 172.21.15.146:0/3008704300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c680c9800 con 0x7f2c680d17f0 2024-04-25T15:22:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c680d26b0 0x7f2c680ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c67fff640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6815ffc0 0x7f2c6815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c677fe640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c681f58a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800d5f0 tx=0x7f2c5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c580125f0 con 0x7f2c680d17f0 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c58019070 con 0x7f2c680d17f0 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c680ced70 con 0x7f2c680d17f0 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c580129d0 con 0x7f2c680d17f0 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.033+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c680cf330 con 0x7f2c680d17f0 2024-04-25T15:22:17.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.034+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6806a0e0 con 0x7f2c680d17f0 2024-04-25T15:22:17.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.035+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f2c58012b70 con 0x7f2c680d17f0 2024-04-25T15:22:17.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.035+0000 7f2c4ffff640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2c54074570 0x7f2c54076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:17.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.035+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2c54074570 0x7f2c54076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:17.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.035+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5249+0+0 (secure 0 0 0) 0x7f2c58095440 con 0x7f2c680d17f0 2024-04-25T15:22:17.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.036+0000 7f2c66ffd640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2c54074570 0x7f2c54076a80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c034000 tx=0x7f2c5c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:17.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.040+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c5805fda0 con 0x7f2c680d17f0 2024-04-25T15:22:17.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:17.215+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2c6809bd60 con 0x7f2c680d17f0 2024-04-25T15:22:18.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.202+0000 7f2c4ffff640 1 -- 172.21.15.146:0/3008704300 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v51) ==== 138+0+0 (secure 0 0 0) 0x7f2c5805f6a0 con 0x7f2c680d17f0 2024-04-25T15:22:18.203 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' created 2024-04-25T15:22:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.206+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2c54074570 msgr2=0x7f2c54076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.206+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2c54074570 0x7f2c54076a80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c034000 tx=0x7f2c5c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.206+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 msgr2=0x7f2c681f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:18.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.206+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c681f58a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800d5f0 tx=0x7f2c5800dab0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.207+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 shutdown_connections 2024-04-25T15:22:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.207+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2c54074570 0x7f2c54076a80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.207+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c6815ffc0 0x7f2c6815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.207+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c680d26b0 0x7f2c680ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.207+0000 7f2c6db9e640 1 --2- 172.21.15.146:0/3008704300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c680d17f0 0x7f2c681f58a0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.207+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 >> 172.21.15.146:0/3008704300 conn(0x7f2c680b7460 msgr2=0x7f2c680b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:18.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.211+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 shutdown_connections 2024-04-25T15:22:18.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.211+0000 7f2c6db9e640 1 -- 172.21.15.146:0/3008704300 wait complete. 2024-04-25T15:22:18.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:22:18.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.345+0000 7fd4eb0d8640 1 Processor -- start 2024-04-25T15:22:18.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.345+0000 7fd4eb0d8640 1 -- start start 2024-04-25T15:22:18.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.346+0000 7fd4eb0d8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4e40b90d0 0x7fd4e40ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.346+0000 7fd4eb0d8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd4e408ca00 0x7fd4e40ced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.346+0000 7fd4eb0d8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4e40b2cb0 con 0x7fd4e4164580 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.346+0000 7fd4eb0d8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e409ccf0 con 0x7fd4e40b90d0 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4eb0d8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e406f110 con 0x7fd4e408ca00 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4e40b90d0 0x7fd4e40ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4e40b90d0 0x7fd4e40ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58148/0 (socket says 172.21.15.146:58148) 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 -- 172.21.15.146:0/2577371576 learned_addr learned my addr 172.21.15.146:0/2577371576 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e3fff640 1 --2- 172.21.15.146:0/2577371576 >> v2:172.21.15.177:3300/0 conn(0x7fd4e408ca00 0x7fd4e40ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 -- 172.21.15.146:0/2577371576 >> v2:172.21.15.177:3300/0 conn(0x7fd4e408ca00 msgr2=0x7fd4e40ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 --2- 172.21.15.146:0/2577371576 >> v2:172.21.15.177:3300/0 conn(0x7fd4e408ca00 0x7fd4e40ced60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 -- 172.21.15.146:0/2577371576 >> v1:172.21.15.146:6789/0 conn(0x7fd4e4164580 legacy=0x7fd4e4164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:18.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.347+0000 7fd4e8e74640 1 -- 172.21.15.146:0/2577371576 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e406f3f0 con 0x7fd4e40b90d0 2024-04-25T15:22:18.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.348+0000 7fd4e8e74640 1 --2- 172.21.15.146:0/2577371576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e40ce820 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b60f0 tx=0x7fd4d402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139dc553e7955c0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:18.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.348+0000 7fd4e37fe640 1 -- 172.21.15.146:0/2577371576 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4037070 con 0x7fd4e40b90d0 2024-04-25T15:22:18.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.348+0000 7fd4e37fe640 1 -- 172.21.15.146:0/2577371576 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d4032440 con 0x7fd4e40b90d0 2024-04-25T15:22:18.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.348+0000 7fd4e37fe640 1 -- 172.21.15.146:0/2577371576 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4032810 con 0x7fd4e40b90d0 2024-04-25T15:22:18.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.348+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/2577371576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 msgr2=0x7fd4e40ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/2577371576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e40ce820 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40b60f0 tx=0x7fd4d402a270 comp rx=0 tx=0).stop 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/2577371576 shutdown_connections 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/2577371576 >> v2:172.21.15.177:3300/0 conn(0x7fd4e408ca00 0x7fd4e40ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/2577371576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e40ce820 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/2577371576 >> 172.21.15.146:0/2577371576 conn(0x7fd4e40b5340 msgr2=0x7fd4e40b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/2577371576 shutdown_connections 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.349+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/2577371576 wait complete. 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.350+0000 7fd4eb0d8640 1 Processor -- start 2024-04-25T15:22:18.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.350+0000 7fd4eb0d8640 1 -- start start 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4eb0d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4e408ca00 0x7fd4e409f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4eb0d8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e41551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4eb0d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4e4164580 0x7fd4e41f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4eb0d8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e4165220 con 0x7fd4e40b90d0 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4eb0d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e41650a0 con 0x7fd4e408ca00 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4eb0d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e41653a0 con 0x7fd4e4164580 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e41551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e41551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58156/0 (socket says 172.21.15.146:58156) 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 -- 172.21.15.146:0/1215837888 learned_addr learned my addr 172.21.15.146:0/1215837888 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e8e74640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4e408ca00 0x7fd4e409f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e9675640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4e4164580 0x7fd4e41f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 -- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4e4164580 msgr2=0x7fd4e41f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:18.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4e4164580 0x7fd4e41f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 -- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4e408ca00 msgr2=0x7fd4e409f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4e408ca00 0x7fd4e409f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 -- 172.21.15.146:0/1215837888 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e4155700 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e9675640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4e4164580 0x7fd4e41f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.352+0000 7fd4e3fff640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e41551c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800c7f0 tx=0x7fd4d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.353+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8015070 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.353+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d8004220 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.353+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e4152d00 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.353+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8007620 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.353+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e41532c0 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.354+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e4152a50 con 0x7fd4e40b90d0 2024-04-25T15:22:18.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.354+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fd4d80077c0 con 0x7fd4e40b90d0 2024-04-25T15:22:18.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.355+0000 7fd4e17fa640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4d0074540 0x7fd4d0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:18.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.355+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5619+0+0 (secure 0 0 0) 0x7fd4d8095490 con 0x7fd4e40b90d0 2024-04-25T15:22:18.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.355+0000 7fd4e8e74640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4d0074540 0x7fd4d0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:18.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.355+0000 7fd4e8e74640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4d0074540 0x7fd4d0076a50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4034000 tx=0x7fd4d402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:18.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.360+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4e4155700 con 0x7fd4e40b90d0 2024-04-25T15:22:18.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:18.536+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd4e40507b0 con 0x7fd4e40b90d0 2024-04-25T15:22:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.204+0000 7fd4e17fa640 1 -- 172.21.15.146:0/1215837888 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v52) ==== 170+0+0 (secure 0 0 0) 0x7fd4d805fb70 con 0x7fd4e40b90d0 2024-04-25T15:22:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' created 2024-04-25T15:22:19.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.208+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4d0074540 msgr2=0x7fd4d0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:19.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.208+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4d0074540 0x7fd4d0076a50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4034000 tx=0x7fd4d402e000 comp rx=0 tx=0).stop 2024-04-25T15:22:19.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.208+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 msgr2=0x7fd4e41551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:19.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.208+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e41551c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800c7f0 tx=0x7fd4d800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.209+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 shutdown_connections 2024-04-25T15:22:19.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.209+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd4d0074540 0x7fd4d0076a50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.209+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4e4164580 0x7fd4e41f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.209+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4e40b90d0 0x7fd4e41551c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.209+0000 7fd4eb0d8640 1 --2- 172.21.15.146:0/1215837888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4e408ca00 0x7fd4e409f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.209+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 >> 172.21.15.146:0/1215837888 conn(0x7fd4e40b5340 msgr2=0x7fd4e4071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:19.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.211+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 shutdown_connections 2024-04-25T15:22:19.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.213+0000 7fd4eb0d8640 1 -- 172.21.15.146:0/1215837888 wait complete. 2024-04-25T15:22:19.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:22:19.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.357+0000 7fc4d53b4640 1 Processor -- start 2024-04-25T15:22:19.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.357+0000 7fc4d53b4640 1 -- start start 2024-04-25T15:22:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4d53b4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc4d0071d30 0x7fc4d00d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4d53b4640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d00cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4d53b4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d00d2260 con 0x7fc4d00b2cd0 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4d53b4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d00d2540 con 0x7fc4d0071d30 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4d53b4640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d00d2820 con 0x7fc4d00d3300 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc4d0071d30 0x7fc4d00d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc4d0071d30 0x7fc4d00d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58164/0 (socket says 172.21.15.146:58164) 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 -- 172.21.15.146:0/2574762050 learned_addr learned my addr 172.21.15.146:0/2574762050 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4cf7fe640 1 --2- 172.21.15.146:0/2574762050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d00cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 -- 172.21.15.146:0/2574762050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 msgr2=0x7fc4d00cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 --2- 172.21.15.146:0/2574762050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d00cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 -- 172.21.15.146:0/2574762050 >> v1:172.21.15.146:6789/0 conn(0x7fc4d00b2cd0 legacy=0x7fc4d0071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.359+0000 7fc4ce7fc640 1 -- 172.21.15.146:0/2574762050 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d00d2b00 con 0x7fc4d0071d30 2024-04-25T15:22:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.360+0000 7fc4ce7fc640 1 --2- 172.21.15.146:0/2574762050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d0071d30 0x7fc4d00d1c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009870 tx=0x7fc4c402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814d8bb08ee802b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.360+0000 7fc4cdffb640 1 -- 172.21.15.146:0/2574762050 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4037070 con 0x7fc4d0071d30 2024-04-25T15:22:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.360+0000 7fc4cdffb640 1 -- 172.21.15.146:0/2574762050 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c4032440 con 0x7fc4d0071d30 2024-04-25T15:22:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.360+0000 7fc4cdffb640 1 -- 172.21.15.146:0/2574762050 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4032810 con 0x7fc4d0071d30 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 -- 172.21.15.146:0/2574762050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d0071d30 msgr2=0x7fc4d00d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/2574762050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d0071d30 0x7fc4d00d1c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009870 tx=0x7fc4c402a1c0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 -- 172.21.15.146:0/2574762050 shutdown_connections 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/2574762050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d00cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/2574762050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d0071d30 0x7fc4d00d1c80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 -- 172.21.15.146:0/2574762050 >> 172.21.15.146:0/2574762050 conn(0x7fc4d00b52e0 msgr2=0x7fc4d00b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 -- 172.21.15.146:0/2574762050 shutdown_connections 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.361+0000 7fc4d53b4640 1 -- 172.21.15.146:0/2574762050 wait complete. 2024-04-25T15:22:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.363+0000 7fc4d53b4640 1 Processor -- start 2024-04-25T15:22:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.363+0000 7fc4d53b4640 1 -- start start 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4d53b4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4d00b2cd0 0x7fc4d0158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4d53b4640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d009d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4d53b4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 0x7fc4d009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4d53b4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d008dbf0 con 0x7fc4d009e2a0 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4d53b4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d00cfb70 con 0x7fc4d00b2cd0 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4d53b4640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d008dd70 con 0x7fc4d00d3300 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 0x7fc4d009e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 0x7fc4d009e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58176/0 (socket says 172.21.15.146:58176) 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 -- 172.21.15.146:0/3527935221 learned_addr learned my addr 172.21.15.146:0/3527935221 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4ce7fc640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d009d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4ceffd640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4d00b2cd0 0x7fc4d0158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 -- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 msgr2=0x7fc4d009d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d009d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 -- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4d00b2cd0 msgr2=0x7fc4d0158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4d00b2cd0 0x7fc4d0158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4cf7fe640 1 -- 172.21.15.146:0/3527935221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d009ebf0 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.364+0000 7fc4ce7fc640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d009d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.365+0000 7fc4cf7fe640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 0x7fc4d009e6b0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800d5a0 tx=0x7fc4c800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.365+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80125f0 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.365+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c8019070 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.365+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d0159100 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.365+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80129f0 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.365+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d00caf80 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.366+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4d006a0e0 con 0x7fc4d009e2a0 2024-04-25T15:22:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.366+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fc4c8012b90 con 0x7fc4d009e2a0 2024-04-25T15:22:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.367+0000 7fc4b77fe640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4bc074540 0x7fc4bc076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.367+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6001+0+0 (secure 0 0 0) 0x7fc4c8095690 con 0x7fc4d009e2a0 2024-04-25T15:22:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.367+0000 7fc4ceffd640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4bc074540 0x7fc4bc076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.367+0000 7fc4ceffd640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4bc074540 0x7fc4bc076a50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00b9490 tx=0x7fc4c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:19.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.372+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc4d009ebf0 con 0x7fc4d009e2a0 2024-04-25T15:22:19.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:19.582+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc4d00b2890 con 0x7fc4d009e2a0 2024-04-25T15:22:20.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.232+0000 7fc4b77fe640 1 -- 172.21.15.146:0/3527935221 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v27) ==== 172+0+0 (secure 0 0 0) 0x7fc4c805fda0 con 0x7fc4d009e2a0 2024-04-25T15:22:20.233 INFO:teuthology.orchestra.run.smithi146.stderr:new fs with metadata pool 7 and data pool 8 2024-04-25T15:22:20.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4bc074540 msgr2=0x7fc4bc076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4bc074540 0x7fc4bc076a50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00b9490 tx=0x7fc4c0009040 comp rx=0 tx=0).stop 2024-04-25T15:22:20.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 msgr2=0x7fc4d009e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 0x7fc4d009e6b0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800d5a0 tx=0x7fc4c800da60 comp rx=0 tx=0).stop 2024-04-25T15:22:20.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 shutdown_connections 2024-04-25T15:22:20.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc4bc074540 0x7fc4bc076a50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4d009e2a0 0x7fc4d009e6b0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc4d00d3300 0x7fc4d009d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 --2- 172.21.15.146:0/3527935221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4d00b2cd0 0x7fc4d0158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.243+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 >> 172.21.15.146:0/3527935221 conn(0x7fc4d00b52e0 msgr2=0x7fc4d00cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.249+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 shutdown_connections 2024-04-25T15:22:20.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.250+0000 7fc4d53b4640 1 -- 172.21.15.146:0/3527935221 wait complete. 2024-04-25T15:22:20.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:20.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.381+0000 7f3a0437d640 1 Processor -- start 2024-04-25T15:22:20.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.381+0000 7f3a0437d640 1 -- start start 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.382+0000 7f3a0437d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f39fc06c550 0x7f39fc06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.382+0000 7f3a0437d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39fc06ca30 con 0x7f39fc0d0e70 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.382+0000 7f3a0437d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39fc06cd10 con 0x7f39fc0d1320 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.382+0000 7f3a0437d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f39fc06cff0 con 0x7f39fc06c550 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.382+0000 7f3a0291a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f39fc0d0e70 0x7f39fc0b8f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38180/0 (socket says 172.21.15.146:38180) 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.382+0000 7f3a0291a640 1 -- 172.21.15.146:0/1457882802 learned_addr learned my addr 172.21.15.146:0/1457882802 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:20.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023095334 0 0) 0x7f39fc06ca30 con 0x7f39fc0d0e70 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a02119640 1 --2- 172.21.15.146:0/1457882802 >> v2:172.21.15.177:3300/0 conn(0x7f39fc06c550 0x7f39fc06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39e8003520 con 0x7f39fc0d0e70 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1712459216 0 0) 0x7f39e8003520 con 0x7f39fc0d0e70 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 >> v2:172.21.15.177:3300/0 conn(0x7f39fc06c550 msgr2=0x7f39fc06dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 --2- 172.21.15.146:0/1457882802 >> v2:172.21.15.177:3300/0 conn(0x7f39fc06c550 0x7f39fc06dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 >> v1:172.21.15.177:6789/0 conn(0x7f39fc0d1320 legacy=0x7f39fc0d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39fc06d2d0 con 0x7f39fc0d0e70 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f39f40019f0 con 0x7f39fc0d0e70 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 >> v1:172.21.15.146:6789/0 conn(0x7f39fc0d0e70 legacy=0x7f39fc0b8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39e80039b0 0x7f39e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39e8004800 0x7f39e8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc06cff0 con 0x7f39e8005250 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f39e800c1e0 con 0x7f39e8004800 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f39e800c3e0 con 0x7f39e80039b0 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a0291a640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a01918640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39e8004800 0x7f39e8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a02119640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39e80039b0 0x7f39e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a0291a640 1 -- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39e80039b0 msgr2=0x7f39e8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a0291a640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39e80039b0 0x7f39e8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a0291a640 1 -- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39e8004800 msgr2=0x7f39e8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a0291a640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39e8004800 0x7f39e8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.383+0000 7f3a0291a640 1 -- 172.21.15.146:0/1457882802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e8003520 con 0x7f39e8005250 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.384+0000 7f3a0291a640 1 -- 172.21.15.146:0/1457882802 reap_dead start 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.384+0000 7f3a0291a640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39e800b9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f39f400a840 tx=0x7f39e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9342947a233bbbdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.384+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f400a460 con 0x7f39e8005250 2024-04-25T15:22:20.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.384+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f402f070 con 0x7f39e8005250 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.384+0000 7f3a01117640 1 -- 172.21.15.146:0/1457882802 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f402bcb0 con 0x7f39e8005250 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 -- 172.21.15.146:0/1457882802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 msgr2=0x7f39e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39e800b9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f39f400a840 tx=0x7f39e8005830 comp rx=0 tx=0).stop 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 -- 172.21.15.146:0/1457882802 shutdown_connections 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1457882802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39e800b9c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 -- 172.21.15.146:0/1457882802 >> 172.21.15.146:0/1457882802 conn(0x7f39fc0d2bc0 msgr2=0x7f39fc0d59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 -- 172.21.15.146:0/1457882802 shutdown_connections 2024-04-25T15:22:20.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.385+0000 7f3a0437d640 1 -- 172.21.15.146:0/1457882802 wait complete. 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.387+0000 7f3a0437d640 1 Processor -- start 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.387+0000 7f3a0437d640 1 -- start start 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0437d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39fc1f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0437d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39fc15b250 0x7f39fc157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0437d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39fc161d70 0x7f39fc162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0437d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc163ee0 con 0x7f39e8005250 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0437d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f39fc163d60 con 0x7f39fc161d70 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0437d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f39fc06dab0 con 0x7f39fc15b250 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a02119640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39fc1f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a02119640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39fc1f5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58200/0 (socket says 172.21.15.146:58200) 2024-04-25T15:22:20.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a02119640 1 -- 172.21.15.146:0/1609181063 learned_addr learned my addr 172.21.15.146:0/1609181063 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a01918640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39fc15b250 0x7f39fc157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a0291a640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39fc161d70 0x7f39fc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a02119640 1 -- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39fc15b250 msgr2=0x7f39fc157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.388+0000 7f3a02119640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39fc15b250 0x7f39fc157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a02119640 1 -- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39fc161d70 msgr2=0x7f39fc162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a02119640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39fc161d70 0x7f39fc162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a02119640 1 -- 172.21.15.146:0/1609181063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39fc15b660 con 0x7f39e8005250 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a0291a640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39fc161d70 0x7f39fc162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a02119640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39fc1f5c10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f39f80118e0 tx=0x7f39f8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f800faf0 con 0x7f39e8005250 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f800fc90 con 0x7f39e8005250 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39fc1626c0 con 0x7f39e8005250 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f8014440 con 0x7f39e8005250 2024-04-25T15:22:20.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.389+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39fc1f6490 con 0x7f39e8005250 2024-04-25T15:22:20.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.390+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39fc06a0e0 con 0x7f39e8005250 2024-04-25T15:22:20.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.390+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f39f8012040 con 0x7f39e8005250 2024-04-25T15:22:20.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.391+0000 7f39f2ffd640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f39e0074540 0x7f39e0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.391+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6101+0+0 (secure 0 0 0) 0x7f39f8095a30 con 0x7f39e8005250 2024-04-25T15:22:20.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.391+0000 7f3a01918640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f39e0074540 0x7f39e0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.391+0000 7f3a01918640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f39e0074540 0x7f39e0076a50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0ca100 tx=0x7f39ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:20.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.396+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39fc1626c0 con 0x7f39e8005250 2024-04-25T15:22:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.570+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39fc15ae20 con 0x7f39e8005250 2024-04-25T15:22:20.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.572+0000 7f39f2ffd640 1 -- 172.21.15.146:0/1609181063 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+17003 (secure 0 0 0) 0x7f39f8019030 con 0x7f39e8005250 2024-04-25T15:22:20.573 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:20.573 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":53,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:19.583084+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f39e0074540 msgr2=0x7f39e0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f39e0074540 0x7f39e0076a50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0ca100 tx=0x7f39ec009040 comp rx=0 tx=0).stop 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 msgr2=0x7f39fc1f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39fc1f5c10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f39f80118e0 tx=0x7f39f8011da0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 shutdown_connections 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f39e0074540 0x7f39e0076a50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39fc161d70 0x7f39fc162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39fc15b250 0x7f39fc157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 --2- 172.21.15.146:0/1609181063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39e8005250 0x7f39fc1f5c10 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 >> 172.21.15.146:0/1609181063 conn(0x7f39fc0d2bc0 msgr2=0x7f39fc164060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:20.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 shutdown_connections 2024-04-25T15:22:20.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.574+0000 7f3a0437d640 1 -- 172.21.15.146:0/1609181063 wait complete. 2024-04-25T15:22:20.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:20.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.693+0000 7f42f70ca640 1 Processor -- start 2024-04-25T15:22:20.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.693+0000 7f42f70ca640 1 -- start start 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f70ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42f00cc280 0x7f42f0070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f70ca640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f00cd490 con 0x7f42f006c8c0 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f70ca640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f00711d0 con 0x7f42f01618f0 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f70ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f42f015bcd0 con 0x7f42f00cc280 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f5667640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f42f006c8c0 0x7f42f006ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50212/0 (socket says 172.21.15.146:50212) 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f5667640 1 -- 172.21.15.146:0/1892833384 learned_addr learned my addr 172.21.15.146:0/1892833384 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.694+0000 7f42f4e66640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42f00cc280 0x7f42f0070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832256029 0 0) 0x7f42f00cd490 con 0x7f42f006c8c0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e0003520 con 0x7f42f006c8c0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26621279 0 0) 0x7f42f00711d0 con 0x7f42f01618f0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f00cd490 con 0x7f42f01618f0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1251242706 0 0) 0x7f42e0003520 con 0x7f42f006c8c0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42f00cc280 msgr2=0x7f42f0070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42f00cc280 0x7f42f0070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 >> v1:172.21.15.177:6789/0 conn(0x7f42f01618f0 legacy=0x7f42f0161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f00cc880 con 0x7f42f006c8c0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f42e80019f0 con 0x7f42f006c8c0 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 >> v1:172.21.15.146:6789/0 conn(0x7f42f006c8c0 legacy=0x7f42f006ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0003b70 0x7f42e0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e0004800 0x7f42e0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42e0005260 0x7f42e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42f015bcd0 con 0x7f42e0004800 2024-04-25T15:22:20.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42e000c1c0 con 0x7f42e0005260 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f42e000c3c0 con 0x7f42e0003b70 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42f4e66640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42f00cc280 0x7f42f0070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42effff640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e0004800 0x7f42e0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42f5667640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42e0005260 0x7f42e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42f4e66640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0003b70 0x7f42e0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42effff640 1 -- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0003b70 msgr2=0x7f42e0003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42effff640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0003b70 0x7f42e0003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42effff640 1 -- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42e0005260 msgr2=0x7f42e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42effff640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42e0005260 0x7f42e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42effff640 1 -- 172.21.15.146:0/1892833384 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e0003520 con 0x7f42e0004800 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42f4e66640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0003b70 0x7f42e0003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.695+0000 7f42f5667640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42e0005260 0x7f42e000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.696+0000 7f42effff640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e0004800 0x7f42e0004c10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f42f00ae960 tx=0x7f42e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223216cd84266647 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:20.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.696+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0014e0 con 0x7f42e0004800 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.696+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42dc0052f0 con 0x7f42e0004800 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.696+0000 7f42ef7fe640 1 -- 172.21.15.146:0/1892833384 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc02abd0 con 0x7f42e0004800 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f70ca640 1 -- 172.21.15.146:0/1892833384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e0004800 msgr2=0x7f42e0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f70ca640 1 --2- 172.21.15.146:0/1892833384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42e0004800 0x7f42e0004c10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f42f00ae960 tx=0x7f42e0005830 comp rx=0 tx=0).stop 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f5667640 1 -- 172.21.15.146:0/1892833384 reap_dead start 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f70ca640 1 -- 172.21.15.146:0/1892833384 shutdown_connections 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f70ca640 1 -- 172.21.15.146:0/1892833384 >> 172.21.15.146:0/1892833384 conn(0x7f42f00ce0c0 msgr2=0x7f42f00ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f70ca640 1 -- 172.21.15.146:0/1892833384 shutdown_connections 2024-04-25T15:22:20.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.697+0000 7f42f70ca640 1 -- 172.21.15.146:0/1892833384 wait complete. 2024-04-25T15:22:20.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.699+0000 7f42f70ca640 1 Processor -- start 2024-04-25T15:22:20.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.699+0000 7f42f70ca640 1 -- start start 2024-04-25T15:22:20.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f70ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0004800 0x7f42f01f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f70ca640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 0x7f42f00b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f70ca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f00b6530 0x7f42f00b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f70ca640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42f00cd120 con 0x7f42f01f8dc0 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f70ca640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42f00cce40 con 0x7f42f00b6530 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f70ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f42f00b0940 con 0x7f42e0004800 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42effff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 0x7f42f00b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42effff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 0x7f42f00b5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33500/0 (socket says 172.21.15.146:33500) 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42effff640 1 -- 172.21.15.146:0/3532084950 learned_addr learned my addr 172.21.15.146:0/3532084950 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.700+0000 7f42f4e66640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0004800 0x7f42f01f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42f5667640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f00b6530 0x7f42f00b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42effff640 1 -- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0004800 msgr2=0x7f42f01f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42effff640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0004800 0x7f42f01f5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42effff640 1 -- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f00b6530 msgr2=0x7f42f00b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42effff640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f00b6530 0x7f42f00b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42effff640 1 -- 172.21.15.146:0/3532084950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f00b4730 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42effff640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 0x7f42f00b5320 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f42dc005040 tx=0x7f42dc0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc007a50 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42dc041070 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f00b6e80 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc02be60 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.701+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f00cf4b0 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.702+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f00cfeb0 con 0x7f42f01f8dc0 2024-04-25T15:22:20.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.703+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f42dc0052f0 con 0x7f42f01f8dc0 2024-04-25T15:22:20.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.703+0000 7f42ed7fa640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42d4074540 0x7f42d4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:20.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.704+0000 7f42f4e66640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42d4074540 0x7f42d4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:20.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.704+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6101+0+0 (secure 0 0 0) 0x7f42dc0b5570 con 0x7f42f01f8dc0 2024-04-25T15:22:20.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.704+0000 7f42f4e66640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42d4074540 0x7f42d4076a50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f42e40090e0 tx=0x7f42e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:20.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.708+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42f00b6e80 con 0x7f42f01f8dc0 2024-04-25T15:22:20.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.884+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42f00b5cc0 con 0x7f42f01f8dc0 2024-04-25T15:22:20.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.885+0000 7f42ed7fa640 1 -- 172.21.15.146:0/3532084950 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+17003 (secure 0 0 0) 0x7f42f00b4730 con 0x7f42f01f8dc0 2024-04-25T15:22:20.886 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:20.886 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":53,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:19.583084+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:20.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42d4074540 msgr2=0x7f42d4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42d4074540 0x7f42d4076a50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f42e40090e0 tx=0x7f42e4008040 comp rx=0 tx=0).stop 2024-04-25T15:22:20.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 msgr2=0x7f42f00b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 0x7f42f00b5320 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f42dc005040 tx=0x7f42dc0012d0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 shutdown_connections 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42d4074540 0x7f42d4076a50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42f00b6530 0x7f42f00b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42f01f8dc0 0x7f42f00b5320 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 --2- 172.21.15.146:0/3532084950 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e0004800 0x7f42f01f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.887+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 >> 172.21.15.146:0/3532084950 conn(0x7f42f00ce0c0 msgr2=0x7f42f00aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.888+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 shutdown_connections 2024-04-25T15:22:20.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:20.888+0000 7f42f70ca640 1 -- 172.21.15.146:0/3532084950 wait complete. 2024-04-25T15:22:20.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:22:21.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.010+0000 7f5add430640 1 Processor -- start 2024-04-25T15:22:21.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.010+0000 7f5add430640 1 -- start start 2024-04-25T15:22:21.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.011+0000 7f5add430640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ad80bb450 0x7f5ad80d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.011+0000 7f5add430640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad80b5230 con 0x7f5ad8163c00 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.011+0000 7f5add430640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad806c8c0 con 0x7f5ad80d0de0 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.011+0000 7f5add430640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5ad806cba0 con 0x7f5ad80bb450 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad6ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5ad8163c00 0x7f5ad806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50216/0 (socket says 172.21.15.146:50216) 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad6ffd640 1 -- 172.21.15.146:0/4196345645 learned_addr learned my addr 172.21.15.146:0/4196345645 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad67fc640 1 --2- 172.21.15.146:0/4196345645 >> v2:172.21.15.146:3300/0 conn(0x7f5ad80bb450 0x7f5ad80d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad67fc640 1 -- 172.21.15.146:0/4196345645 >> v1:172.21.15.177:6789/0 conn(0x7f5ad80d0de0 legacy=0x7f5ad80b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad67fc640 1 -- 172.21.15.146:0/4196345645 >> v1:172.21.15.146:6789/0 conn(0x7f5ad8163c00 legacy=0x7f5ad806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad5ffb640 1 -- 172.21.15.146:0/4196345645 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816978291 0 0) 0x7f5ad806c8c0 con 0x7f5ad80d0de0 2024-04-25T15:22:21.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.012+0000 7f5ad67fc640 1 -- 172.21.15.146:0/4196345645 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad806ce80 con 0x7f5ad80bb450 2024-04-25T15:22:21.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.013+0000 7f5ad67fc640 1 --2- 172.21.15.146:0/4196345645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad80d08a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00097d0 tx=0x7f5ac002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d897266ff7d17f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.013+0000 7f5ad5ffb640 1 -- 172.21.15.146:0/4196345645 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0037070 con 0x7f5ad80bb450 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.013+0000 7f5ad5ffb640 1 -- 172.21.15.146:0/4196345645 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac0032440 con 0x7f5ad80bb450 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.013+0000 7f5ad5ffb640 1 -- 172.21.15.146:0/4196345645 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0032810 con 0x7f5ad80bb450 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 -- 172.21.15.146:0/4196345645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 msgr2=0x7f5ad80d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 --2- 172.21.15.146:0/4196345645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad80d08a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00097d0 tx=0x7f5ac002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 -- 172.21.15.146:0/4196345645 shutdown_connections 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 --2- 172.21.15.146:0/4196345645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad80d08a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 -- 172.21.15.146:0/4196345645 >> 172.21.15.146:0/4196345645 conn(0x7f5ad80b7970 msgr2=0x7f5ad80b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 -- 172.21.15.146:0/4196345645 shutdown_connections 2024-04-25T15:22:21.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.014+0000 7f5add430640 1 -- 172.21.15.146:0/4196345645 wait complete. 2024-04-25T15:22:21.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.016+0000 7f5add430640 1 Processor -- start 2024-04-25T15:22:21.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.016+0000 7f5add430640 1 -- start start 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5add430640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad815c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5add430640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d0de0 0x7f5ad814ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5add430640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad8163c00 0x7f5ad8147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5add430640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80bb860 con 0x7f5ad80bb450 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5add430640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad806c550 con 0x7f5ad80d0de0 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5add430640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad80bb9e0 con 0x7f5ad8163c00 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad815c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad815c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33530/0 (socket says 172.21.15.146:33530) 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 -- 172.21.15.146:0/3830824617 learned_addr learned my addr 172.21.15.146:0/3830824617 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 -- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad8163c00 msgr2=0x7f5ad8147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad77fe640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad8163c00 0x7f5ad8147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad67fc640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d0de0 0x7f5ad814ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad8163c00 0x7f5ad8147940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:21.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 -- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d0de0 msgr2=0x7f5ad814ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d0de0 0x7f5ad814ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 -- 172.21.15.146:0/3830824617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad814b2d0 con 0x7f5ad80bb450 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.017+0000 7f5ad6ffd640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad815c990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000cbf0 tx=0x7f5ad000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.018+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0017070 con 0x7f5ad80bb450 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.018+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad000aca0 con 0x7f5ad80bb450 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.018+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad81488e0 con 0x7f5ad80bb450 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.018+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0012600 con 0x7f5ad80bb450 2024-04-25T15:22:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.018+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad8148ea0 con 0x7f5ad80bb450 2024-04-25T15:22:21.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.019+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad806a0e0 con 0x7f5ad80bb450 2024-04-25T15:22:21.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.019+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f5ad001a070 con 0x7f5ad80bb450 2024-04-25T15:22:21.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.020+0000 7f5acf7fe640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ab8074570 0x7f5ab8076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:21.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.020+0000 7f5ad67fc640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ab8074570 0x7f5ab8076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:21.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.020+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6101+0+0 (secure 0 0 0) 0x7f5ad0095070 con 0x7f5ad80bb450 2024-04-25T15:22:21.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.020+0000 7f5ad67fc640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ab8074570 0x7f5ab8076a80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0034000 tx=0x7f5ac002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:21.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.024+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ad005f030 con 0x7f5ad80bb450 2024-04-25T15:22:21.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:21.235+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5ad80d3ed0 con 0x7f5ad80bb450 2024-04-25T15:22:22.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.245+0000 7f5acf7fe640 1 -- 172.21.15.146:0/3830824617 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f5ad0063330 con 0x7f5ad80bb450 2024-04-25T15:22:22.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ab8074570 msgr2=0x7f5ab8076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:22.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ab8074570 0x7f5ab8076a80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0034000 tx=0x7f5ac002e000 comp rx=0 tx=0).stop 2024-04-25T15:22:22.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 msgr2=0x7f5ad815c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad815c990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000cbf0 tx=0x7f5ad000a060 comp rx=0 tx=0).stop 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 shutdown_connections 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5ab8074570 0x7f5ab8076a80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad8163c00 0x7f5ad8147940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d0de0 0x7f5ad814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 --2- 172.21.15.146:0/3830824617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad80bb450 0x7f5ad815c990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.247+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 >> 172.21.15.146:0/3830824617 conn(0x7f5ad80b7970 msgr2=0x7f5ad80d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.248+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 shutdown_connections 2024-04-25T15:22:22.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.248+0000 7f5add430640 1 -- 172.21.15.146:0/3830824617 wait complete. 2024-04-25T15:22:22.262 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:22:22.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:22:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.378+0000 7f4f35972640 1 Processor -- start 2024-04-25T15:22:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.378+0000 7f4f35972640 1 -- start start 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f35972640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4f3009b5e0 0x7f4f3009b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f35972640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4f300cd290 0x7f4f300cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f35972640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f3009bf30 con 0x7f4f301602e0 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f35972640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4f301606f0 con 0x7f4f3009b5e0 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f35972640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f4f300d2480 con 0x7f4f300cd290 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2e7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4f3009b5e0 0x7f4f3009b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:22.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2e7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4f3009b5e0 0x7f4f3009b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33536/0 (socket says 172.21.15.146:33536) 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3174029897 learned_addr learned my addr 172.21.15.146:0/3174029897 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3174029897 >> v2:172.21.15.177:3300/0 conn(0x7f4f300cd290 msgr2=0x7f4f300cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2f7fe640 1 --2- 172.21.15.146:0/3174029897 >> v2:172.21.15.177:3300/0 conn(0x7f4f300cd290 0x7f4f300cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2e7fc640 1 --2- 172.21.15.146:0/3174029897 >> v2:172.21.15.177:3300/0 conn(0x7f4f300cd290 0x7f4f300cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.379+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3174029897 >> v1:172.21.15.177:6789/0 conn(0x7f4f301602e0 legacy=0x7f4f300b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.380+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3174029897 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f300d2740 con 0x7f4f3009b5e0 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.380+0000 7f4f2e7fc640 1 --2- 172.21.15.146:0/3174029897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f3009b5e0 0x7f4f3009b9f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4f3006f500 tx=0x7f4f2402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7269973b1fe70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.380+0000 7f4f2dffb640 1 -- 172.21.15.146:0/3174029897 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24037070 con 0x7f4f3009b5e0 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.380+0000 7f4f2dffb640 1 -- 172.21.15.146:0/3174029897 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f24032440 con 0x7f4f3009b5e0 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.380+0000 7f4f2dffb640 1 -- 172.21.15.146:0/3174029897 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24032810 con 0x7f4f3009b5e0 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 -- 172.21.15.146:0/3174029897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f3009b5e0 msgr2=0x7f4f3009b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 --2- 172.21.15.146:0/3174029897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f3009b5e0 0x7f4f3009b9f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4f3006f500 tx=0x7f4f2402a230 comp rx=0 tx=0).stop 2024-04-25T15:22:22.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 -- 172.21.15.146:0/3174029897 shutdown_connections 2024-04-25T15:22:22.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 --2- 172.21.15.146:0/3174029897 >> v2:172.21.15.177:3300/0 conn(0x7f4f300cd290 0x7f4f300cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 --2- 172.21.15.146:0/3174029897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f3009b5e0 0x7f4f3009b9f0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 -- 172.21.15.146:0/3174029897 >> 172.21.15.146:0/3174029897 conn(0x7f4f300b7490 msgr2=0x7f4f300b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:22.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 -- 172.21.15.146:0/3174029897 shutdown_connections 2024-04-25T15:22:22.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.381+0000 7f4f35972640 1 -- 172.21.15.146:0/3174029897 wait complete. 2024-04-25T15:22:22.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.383+0000 7f4f35972640 1 Processor -- start 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.383+0000 7f4f35972640 1 -- start start 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f35972640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3009b5e0 0x7f4f300c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f35972640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 0x7f4f300cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f35972640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f301602e0 0x7f4f300c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f35972640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300d2fe0 con 0x7f4f300cd290 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f35972640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300d2d00 con 0x7f4f301602e0 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f35972640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3006f7f0 con 0x7f4f3009b5e0 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f2e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 0x7f4f300cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:22.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f2e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 0x7f4f300cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33546/0 (socket says 172.21.15.146:33546) 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.384+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3152811979 learned_addr learned my addr 172.21.15.146:0/3152811979 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2f7fe640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f301602e0 0x7f4f300c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2effd640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3009b5e0 0x7f4f300c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3009b5e0 msgr2=0x7f4f300c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2e7fc640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3009b5e0 0x7f4f300c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f301602e0 msgr2=0x7f4f300c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2e7fc640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f301602e0 0x7f4f300c7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2e7fc640 1 -- 172.21.15.146:0/3152811979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f301534d0 con 0x7f4f300cd290 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f2e7fc640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 0x7f4f300cc0e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4f24000c00 tx=0x7f4f240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24041070 con 0x7f4f300cd290 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2403c870 con 0x7f4f300cd290 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f300c6f00 con 0x7f4f300cd290 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24037040 con 0x7f4f300cd290 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.385+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f300c74c0 con 0x7f4f300cd290 2024-04-25T15:22:22.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.386+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f300c9820 con 0x7f4f300cd290 2024-04-25T15:22:22.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.387+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f4f24044070 con 0x7f4f300cd290 2024-04-25T15:22:22.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.387+0000 7f4f17fff640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4f1c074540 0x7f4f1c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:22.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.388+0000 7f4f2effd640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4f1c074540 0x7f4f1c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:22.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.388+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f4f240b5f90 con 0x7f4f300cd290 2024-04-25T15:22:22.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.388+0000 7f4f2effd640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4f1c074540 0x7f4f1c076a50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4f300d1120 tx=0x7f4f2000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:22.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.392+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f301534d0 con 0x7f4f300cd290 2024-04-25T15:22:22.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:22.603+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f4f300c6c90 con 0x7f4f300cd290 2024-04-25T15:22:23.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.245+0000 7f4f17fff640 1 -- 172.21.15.146:0/3152811979 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f4f3009bf30 con 0x7f4f300cd290 2024-04-25T15:22:23.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.247+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4f1c074540 msgr2=0x7f4f1c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.247+0000 7f4f35972640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4f1c074540 0x7f4f1c076a50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4f300d1120 tx=0x7f4f2000d040 comp rx=0 tx=0).stop 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 msgr2=0x7f4f300cc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 0x7f4f300cc0e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4f24000c00 tx=0x7f4f240046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 shutdown_connections 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4f1c074540 0x7f4f1c076a50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f301602e0 0x7f4f300c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f300cd290 0x7f4f300cc0e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 --2- 172.21.15.146:0/3152811979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3009b5e0 0x7f4f300c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 >> 172.21.15.146:0/3152811979 conn(0x7f4f300b7490 msgr2=0x7f4f3015f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 shutdown_connections 2024-04-25T15:22:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.248+0000 7f4f35972640 1 -- 172.21.15.146:0/3152811979 wait complete. 2024-04-25T15:22:23.261 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:22:23.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:23.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.369+0000 7f32e4d34640 1 Processor -- start 2024-04-25T15:22:23.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.369+0000 7f32e4d34640 1 -- start start 2024-04-25T15:22:23.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.370+0000 7f32e4d34640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f32e00d4f60 0x7f32e00d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.370+0000 7f32e4d34640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e00b9200 con 0x7f32e006ccc0 2024-04-25T15:22:23.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.370+0000 7f32e4d34640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e00d1070 con 0x7f32e00d0020 2024-04-25T15:22:23.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.370+0000 7f32e4d34640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f32e006c550 con 0x7f32e00d4f60 2024-04-25T15:22:23.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.370+0000 7f32ded9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f32e006ccc0 0x7f32e006d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50228/0 (socket says 172.21.15.146:50228) 2024-04-25T15:22:23.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.370+0000 7f32ded9d640 1 -- 172.21.15.146:0/2387562276 learned_addr learned my addr 172.21.15.146:0/2387562276 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 --2- 172.21.15.146:0/2387562276 >> v2:172.21.15.177:3300/0 conn(0x7f32e00d4f60 0x7f32e00d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3190593617 0 0) 0x7f32d8001490 con 0x7f32e006ccc0 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32cc003520 con 0x7f32e006ccc0 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1982322965 0 0) 0x7f32cc003520 con 0x7f32e006ccc0 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 >> v2:172.21.15.177:3300/0 conn(0x7f32e00d4f60 msgr2=0x7f32e00d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 --2- 172.21.15.146:0/2387562276 >> v2:172.21.15.177:3300/0 conn(0x7f32e00d4f60 0x7f32e00d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 >> v1:172.21.15.177:6789/0 conn(0x7f32e00d0020 legacy=0x7f32e00d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e00d1310 con 0x7f32e006ccc0 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f32d8001bc0 con 0x7f32e006ccc0 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 >> v1:172.21.15.146:6789/0 conn(0x7f32e006ccc0 legacy=0x7f32e006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc004800 0x7f32cc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32cc005350 0x7f32cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f32e006c550 con 0x7f32cc0039b0 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc00c280 con 0x7f32cc004800 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f32cc00c480 con 0x7f32cc005350 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32d7fff640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc004800 0x7f32cc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ded9d640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32cc005350 0x7f32cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 -- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32cc005350 msgr2=0x7f32cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32cc005350 0x7f32cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 -- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc004800 msgr2=0x7f32cc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32cc004800 0x7f32cc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32de59c640 1 -- 172.21.15.146:0/2387562276 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc003520 con 0x7f32cc0039b0 2024-04-25T15:22:23.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.371+0000 7f32ded9d640 1 -- 172.21.15.146:0/2387562276 reap_dead start 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.372+0000 7f32de59c640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32cc003e20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f32e0163ce0 tx=0x7f32cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7064016629a15503 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.372+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0036070 con 0x7f32cc0039b0 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.372+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d0031400 con 0x7f32cc0039b0 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.372+0000 7f32ddd9b640 1 -- 172.21.15.146:0/2387562276 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0009690 con 0x7f32cc0039b0 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 -- 172.21.15.146:0/2387562276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 msgr2=0x7f32cc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32cc003e20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f32e0163ce0 tx=0x7f32cc005830 comp rx=0 tx=0).stop 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 -- 172.21.15.146:0/2387562276 shutdown_connections 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2387562276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32cc003e20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 -- 172.21.15.146:0/2387562276 >> 172.21.15.146:0/2387562276 conn(0x7f32e00d2450 msgr2=0x7f32e00d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 -- 172.21.15.146:0/2387562276 shutdown_connections 2024-04-25T15:22:23.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.373+0000 7f32e4d34640 1 -- 172.21.15.146:0/2387562276 wait complete. 2024-04-25T15:22:23.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.375+0000 7f32e4d34640 1 Processor -- start 2024-04-25T15:22:23.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.375+0000 7f32e4d34640 1 -- start start 2024-04-25T15:22:23.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32e4d34640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32e01e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32e4d34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32e0159ad0 0x7f32e0159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32e4d34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e015a020 0x7f32e01f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32e4d34640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f32e00b5150 con 0x7f32cc0039b0 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32e4d34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32e006c830 con 0x7f32e015a020 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32e4d34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f32e00b52d0 con 0x7f32e0159ad0 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32de59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32e01e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32de59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32e01e9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33560/0 (socket says 172.21.15.146:33560) 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32de59c640 1 -- 172.21.15.146:0/2757486946 learned_addr learned my addr 172.21.15.146:0/2757486946 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.376+0000 7f32d7fff640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32e0159ad0 0x7f32e0159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32ded9d640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e015a020 0x7f32e01f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32de59c640 1 -- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32e0159ad0 msgr2=0x7f32e0159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32de59c640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32e0159ad0 0x7f32e0159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32de59c640 1 -- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e015a020 msgr2=0x7f32e01f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32de59c640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e015a020 0x7f32e01f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32de59c640 1 -- 172.21.15.146:0/2757486946 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e00c97e0 con 0x7f32cc0039b0 2024-04-25T15:22:23.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32ded9d640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e015a020 0x7f32e01f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32de59c640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32e01e9df0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009b80 tx=0x7f32d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0040020 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d003b680 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0036040 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e01e9030 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.377+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e01e95f0 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.378+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e006a0e0 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.379+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f32d0043020 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.379+0000 7f32d6ffd640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32c0074570 0x7f32c0076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.379+0000 7f32d7fff640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32c0074570 0x7f32c0076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.380+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f32d00b5a00 con 0x7f32cc0039b0 2024-04-25T15:22:23.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.380+0000 7f32d7fff640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32c0074570 0x7f32c0076a80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f32e01eae80 tx=0x7f32c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:23.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.384+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f32d004a030 con 0x7f32cc0039b0 2024-04-25T15:22:23.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.596+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32e00bb870 con 0x7f32cc0039b0 2024-04-25T15:22:23.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.596+0000 7f32d6ffd640 1 -- 172.21.15.146:0/2757486946 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5187 (secure 0 0 0) 0x7f32d007fa50 con 0x7f32cc0039b0 2024-04-25T15:22:23.599 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:23.599 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:22.605197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/865508848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":865508848},{"type":"v1","addr":"172.21.15.146:6836","nonce":865508848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":3}]} 2024-04-25T15:22:23.599 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 30 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32c0074570 msgr2=0x7f32c0076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32c0074570 0x7f32c0076a80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f32e01eae80 tx=0x7f32c8009040 comp rx=0 tx=0).stop 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 msgr2=0x7f32e01e9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32e01e9df0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009b80 tx=0x7f32d0004730 comp rx=0 tx=0).stop 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 shutdown_connections 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f32c0074570 0x7f32c0076a80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32e015a020 0x7f32e01f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32e0159ad0 0x7f32e0159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 --2- 172.21.15.146:0/2757486946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32cc0039b0 0x7f32e01e9df0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 >> 172.21.15.146:0/2757486946 conn(0x7f32e00d2450 msgr2=0x7f32e015bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:23.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 shutdown_connections 2024-04-25T15:22:23.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.598+0000 7f32e4d34640 1 -- 172.21.15.146:0/2757486946 wait complete. 2024-04-25T15:22:23.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:23.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.717+0000 7fa09386e640 1 Processor -- start 2024-04-25T15:22:23.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.717+0000 7fa09386e640 1 -- start start 2024-04-25T15:22:23.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa09386e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa09386e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa08c0cfa90 0x7fa08c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa09386e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c0cfea0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa09386e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa08c0b4f20 con 0x7fa08c0cfa90 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa09386e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa08c0b3a70 con 0x7fa08c0b4b10 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa08c0cfa90 0x7fa08c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa08c0cfa90 0x7fa08c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33566/0 (socket says 172.21.15.146:33566) 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 -- 172.21.15.146:0/325106193 learned_addr learned my addr 172.21.15.146:0/325106193 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa09160a640 1 --2- 172.21.15.146:0/325106193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 -- 172.21.15.146:0/325106193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 msgr2=0x7fa08c0cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 --2- 172.21.15.146:0/325106193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c0cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 -- 172.21.15.146:0/325106193 >> v1:172.21.15.177:6789/0 conn(0x7fa08c0d2c50 legacy=0x7fa08c0af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa083fff640 1 -- 172.21.15.146:0/325106193 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749186914 0 0) 0x7fa084001490 con 0x7fa08c0d2c50 2024-04-25T15:22:23.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.719+0000 7fa090e09640 1 -- 172.21.15.146:0/325106193 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c0b3d50 con 0x7fa08c0cfa90 2024-04-25T15:22:23.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.720+0000 7fa090e09640 1 --2- 172.21.15.146:0/325106193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0d2710 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009870 tx=0x7fa07c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22493c7063196ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:23.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.720+0000 7fa083fff640 1 -- 172.21.15.146:0/325106193 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c037070 con 0x7fa08c0cfa90 2024-04-25T15:22:23.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.720+0000 7fa083fff640 1 -- 172.21.15.146:0/325106193 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07c032440 con 0x7fa08c0cfa90 2024-04-25T15:22:23.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.720+0000 7fa083fff640 1 -- 172.21.15.146:0/325106193 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c032810 con 0x7fa08c0cfa90 2024-04-25T15:22:23.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 -- 172.21.15.146:0/325106193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0cfa90 msgr2=0x7fa08c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 --2- 172.21.15.146:0/325106193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0d2710 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009870 tx=0x7fa07c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 -- 172.21.15.146:0/325106193 shutdown_connections 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 --2- 172.21.15.146:0/325106193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0d2710 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 --2- 172.21.15.146:0/325106193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c0cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 -- 172.21.15.146:0/325106193 >> 172.21.15.146:0/325106193 conn(0x7fa08c0b73b0 msgr2=0x7fa08c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 -- 172.21.15.146:0/325106193 shutdown_connections 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.721+0000 7fa09386e640 1 -- 172.21.15.146:0/325106193 wait complete. 2024-04-25T15:22:23.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.723+0000 7fa09386e640 1 Processor -- start 2024-04-25T15:22:23.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.723+0000 7fa09386e640 1 -- start start 2024-04-25T15:22:23.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09386e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09386e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09386e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 0x7fa08c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09386e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c0b45f0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09386e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c0b4310 con 0x7fa08c0cfa90 2024-04-25T15:22:23.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09386e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa08c0d3060 con 0x7fa08c0b4b10 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 0x7fa08c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 0x7fa08c0cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33574/0 (socket says 172.21.15.146:33574) 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 -- 172.21.15.146:0/1951886033 learned_addr learned my addr 172.21.15.146:0/1951886033 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09160a640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa090e09640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 -- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 msgr2=0x7fa08c1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c1556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 -- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08c0cfa90 msgr2=0x7fa08c0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa091e0b640 1 -- 172.21.15.146:0/1951886033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c15cfe0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.724+0000 7fa09160a640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.725+0000 7fa091e0b640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 0x7fa08c0cb430 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c780 tx=0x7fa08400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.725+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084003b70 con 0x7fa08c0d2c50 2024-04-25T15:22:23.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.725+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa084003d30 con 0x7fa08c0d2c50 2024-04-25T15:22:23.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.725+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0840052b0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.725+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c1f62f0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.725+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08c1f68b0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.726+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08c06a0e0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.727+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fa084005450 con 0x7fa08c0d2c50 2024-04-25T15:22:23.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.727+0000 7fa081ffb640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa070074570 0x7fa070076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:23.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.727+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7fa084095150 con 0x7fa08c0d2c50 2024-04-25T15:22:23.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.727+0000 7fa09160a640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa070074570 0x7fa070076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:23.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.728+0000 7fa09160a640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa070074570 0x7fa070076a80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0cd1b0 tx=0x7fa088008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:23.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.732+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa08c0cfea0 con 0x7fa08c0d2c50 2024-04-25T15:22:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.907+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa08c0b2f00 con 0x7fa08c0d2c50 2024-04-25T15:22:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.908+0000 7fa081ffb640 1 -- 172.21.15.146:0/1951886033 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+16990 (secure 0 0 0) 0x7fa08405f880 con 0x7fa08c0d2c50 2024-04-25T15:22:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:23.911 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":54,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:21.232449+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa070074570 msgr2=0x7fa070076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa070074570 0x7fa070076a80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0cd1b0 tx=0x7fa088008040 comp rx=0 tx=0).stop 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 msgr2=0x7fa08c0cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 0x7fa08c0cb430 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c780 tx=0x7fa08400cc40 comp rx=0 tx=0).stop 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 shutdown_connections 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa070074570 0x7fa070076a80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.910+0000 7fa09386e640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08c0d2c50 0x7fa08c0cb430 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.911+0000 7fa09386e640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08c0cfa90 0x7fa08c0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.911+0000 7fa09386e640 1 --2- 172.21.15.146:0/1951886033 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa08c0b4b10 0x7fa08c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.911+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 >> 172.21.15.146:0/1951886033 conn(0x7fa08c0b73b0 msgr2=0x7fa08c06ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.911+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 shutdown_connections 2024-04-25T15:22:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:23.911+0000 7fa09386e640 1 -- 172.21.15.146:0/1951886033 wait complete. 2024-04-25T15:22:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:22:23.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:22:24.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.030+0000 7f92a22ca640 1 Processor -- start 2024-04-25T15:22:24.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.030+0000 7f92a22ca640 1 -- start start 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a22ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d0730 0x7f929c0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a22ca640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f929c06ccc0 0x7f929c06d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a22ca640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f929c0b9200 con 0x7f929c0d4f60 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a22ca640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f929c06c550 con 0x7f929c06ccc0 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a22ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f929c06c830 con 0x7f929c0d0730 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f929b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d0730 0x7f929c0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f929c06ccc0 0x7f929c06d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f929c06ccc0 0x7f929c06d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47782/0 (socket says 172.21.15.146:47782) 2024-04-25T15:22:24.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f929bfff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f929c0d4f60 0x7f929c0d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:33320/0 (socket says 172.21.15.146:33320) 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f929b7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d0730 0x7f929c0d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:37926/0 (socket says 172.21.15.146:37926) 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 -- 172.21.15.146:0/3570046851 learned_addr learned my addr 172.21.15.146:0/3570046851 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 -- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d0730 msgr2=0x7f929c0d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 --2- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d0730 0x7f929c0d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 -- 172.21.15.146:0/3570046851 >> v1:172.21.15.177:6789/0 conn(0x7f929c0d4f60 legacy=0x7f929c0d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f929affd640 1 -- 172.21.15.146:0/3570046851 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4211735943 0 0) 0x7f928c001110 con 0x7f929c0d4f60 2024-04-25T15:22:24.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.032+0000 7f92a0867640 1 -- 172.21.15.146:0/3570046851 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c0d1070 con 0x7f929c06ccc0 2024-04-25T15:22:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.033+0000 7f92a0867640 1 --2- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c06ccc0 0x7f929c06d0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9294009e50 tx=0x7f929402a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b83072bd88baed66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.033+0000 7f929affd640 1 -- 172.21.15.146:0/3570046851 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294036070 con 0x7f929c06ccc0 2024-04-25T15:22:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.033+0000 7f929affd640 1 -- 172.21.15.146:0/3570046851 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9294031600 con 0x7f929c06ccc0 2024-04-25T15:22:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.033+0000 7f929affd640 1 -- 172.21.15.146:0/3570046851 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940319d0 con 0x7f929c06ccc0 2024-04-25T15:22:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 -- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c06ccc0 msgr2=0x7f929c06d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 --2- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c06ccc0 0x7f929c06d0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9294009e50 tx=0x7f929402a5f0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 -- 172.21.15.146:0/3570046851 shutdown_connections 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 --2- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c06ccc0 0x7f929c06d0d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 --2- 172.21.15.146:0/3570046851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d0730 0x7f929c0d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 -- 172.21.15.146:0/3570046851 >> 172.21.15.146:0/3570046851 conn(0x7f929c0d2450 msgr2=0x7f929c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 -- 172.21.15.146:0/3570046851 shutdown_connections 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.034+0000 7f92a22ca640 1 -- 172.21.15.146:0/3570046851 wait complete. 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.036+0000 7f92a22ca640 1 Processor -- start 2024-04-25T15:22:24.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.036+0000 7f92a22ca640 1 -- start start 2024-04-25T15:22:24.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a22ca640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 0x7f929c0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a22ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d4f60 0x7f929c14ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a22ca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c162140 0x7f929c15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a22ca640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f929c0b5150 con 0x7f929c06ccc0 2024-04-25T15:22:24.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a22ca640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f929c0d15f0 con 0x7f929c162140 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a22ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f929c0b52d0 con 0x7f929c0d4f60 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f929bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 0x7f929c0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f929bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 0x7f929c0c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33582/0 (socket says 172.21.15.146:33582) 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f929bfff640 1 -- 172.21.15.146:0/1695567697 learned_addr learned my addr 172.21.15.146:0/1695567697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f929b7fe640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d4f60 0x7f929c14ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.037+0000 7f92a0867640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c162140 0x7f929c15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f929bfff640 1 -- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d4f60 msgr2=0x7f929c14ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f929bfff640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d4f60 0x7f929c14ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f929bfff640 1 -- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c162140 msgr2=0x7f929c15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f929bfff640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c162140 0x7f929c15e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f929bfff640 1 -- 172.21.15.146:0/1695567697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c162920 con 0x7f929c06ccc0 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f929bfff640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 0x7f929c0c97f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c3e0 tx=0x7f928c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c0074e0 con 0x7f929c06ccc0 2024-04-25T15:22:24.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c0047e0 con 0x7f929c06ccc0 2024-04-25T15:22:24.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c004bd0 con 0x7f929c06ccc0 2024-04-25T15:22:24.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c162c00 con 0x7f929c06ccc0 2024-04-25T15:22:24.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.038+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c1631c0 con 0x7f929c06ccc0 2024-04-25T15:22:24.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.039+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c06a0e0 con 0x7f929c06ccc0 2024-04-25T15:22:24.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.040+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f928c011070 con 0x7f929c06ccc0 2024-04-25T15:22:24.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.040+0000 7f9298ff9640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9288074540 0x7f9288076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.040+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f928c094e30 con 0x7f929c06ccc0 2024-04-25T15:22:24.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.040+0000 7f929b7fe640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9288074540 0x7f9288076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.041+0000 7f929b7fe640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9288074540 0x7f9288076a50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f929c0c8e10 tx=0x7f9290009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.045+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f929c0b9200 con 0x7f929c06ccc0 2024-04-25T15:22:24.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.216+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f929c0d4100 con 0x7f9288074540 2024-04-25T15:22:24.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.223+0000 7f9298ff9640 1 -- 172.21.15.146:0/1695567697 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f929c0d4100 con 0x7f9288074540 2024-04-25T15:22:24.228 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:22:24.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9288074540 msgr2=0x7f9288076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9288074540 0x7f9288076a50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f929c0c8e10 tx=0x7f9290009190 comp rx=0 tx=0).stop 2024-04-25T15:22:24.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 msgr2=0x7f929c0c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 0x7f929c0c97f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c3e0 tx=0x7f928c00c8a0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 shutdown_connections 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9288074540 0x7f9288076a50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c162140 0x7f929c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f929c0d4f60 0x7f929c14ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 --2- 172.21.15.146:0/1695567697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f929c06ccc0 0x7f929c0c97f0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 >> 172.21.15.146:0/1695567697 conn(0x7f929c0d2450 msgr2=0x7f929c0b6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 shutdown_connections 2024-04-25T15:22:24.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.226+0000 7f92a22ca640 1 -- 172.21.15.146:0/1695567697 wait complete. 2024-04-25T15:22:24.239 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-25T15:22:24.020691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222448,"kb_used_data":6248,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752272,"statfs":{"total":773094113280,"available":772866326528,"internally_reserved":0,"allocated":6397952,"data_stored":3507156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.817111"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837679+0000","last_change":"2024-04-25T15:22:20.247164+0000","last_active":"2024-04-25T15:22:21.837679+0000","last_peered":"2024-04-25T15:22:21.837679+0000","last_clean":"2024-04-25T15:22:21.837679+0000","last_became_active":"2024-04-25T15:22:20.246108+0000","last_became_peered":"2024-04-25T15:22:20.246108+0000","last_unstale":"2024-04-25T15:22:21.837679+0000","last_undegraded":"2024-04-25T15:22:21.837679+0000","last_fullsized":"2024-04-25T15:22:21.837679+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:11.666514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245214+0000","last_change":"2024-04-25T15:22:20.237989+0000","last_active":"2024-04-25T15:22:21.245214+0000","last_peered":"2024-04-25T15:22:21.245214+0000","last_clean":"2024-04-25T15:22:21.245214+0000","last_became_active":"2024-04-25T15:22:20.237719+0000","last_became_peered":"2024-04-25T15:22:20.237719+0000","last_unstale":"2024-04-25T15:22:21.245214+0000","last_undegraded":"2024-04-25T15:22:21.245214+0000","last_fullsized":"2024-04-25T15:22:21.245214+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:56.474098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245742+0000","last_change":"2024-04-25T15:22:20.240447+0000","last_active":"2024-04-25T15:22:21.245742+0000","last_peered":"2024-04-25T15:22:21.245742+0000","last_clean":"2024-04-25T15:22:21.245742+0000","last_became_active":"2024-04-25T15:22:20.240100+0000","last_became_peered":"2024-04-25T15:22:20.240100+0000","last_unstale":"2024-04-25T15:22:21.245742+0000","last_undegraded":"2024-04-25T15:22:21.245742+0000","last_fullsized":"2024-04-25T15:22:21.245742+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:13.143871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245312+0000","last_change":"2024-04-25T15:22:20.252667+0000","last_active":"2024-04-25T15:22:21.245312+0000","last_peered":"2024-04-25T15:22:21.245312+0000","last_clean":"2024-04-25T15:22:21.245312+0000","last_became_active":"2024-04-25T15:22:20.252363+0000","last_became_peered":"2024-04-25T15:22:20.252363+0000","last_unstale":"2024-04-25T15:22:21.245312+0000","last_undegraded":"2024-04-25T15:22:21.245312+0000","last_fullsized":"2024-04-25T15:22:21.245312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:04.130563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838181+0000","last_change":"2024-04-25T15:22:20.245860+0000","last_active":"2024-04-25T15:22:21.838181+0000","last_peered":"2024-04-25T15:22:21.838181+0000","last_clean":"2024-04-25T15:22:21.838181+0000","last_became_active":"2024-04-25T15:22:20.245248+0000","last_became_peered":"2024-04-25T15:22:20.245248+0000","last_unstale":"2024-04-25T15:22:21.838181+0000","last_undegraded":"2024-04-25T15:22:21.838181+0000","last_fullsized":"2024-04-25T15:22:21.838181+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:16.705766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837686+0000","last_change":"2024-04-25T15:22:20.250097+0000","last_active":"2024-04-25T15:22:21.837686+0000","last_peered":"2024-04-25T15:22:21.837686+0000","last_clean":"2024-04-25T15:22:21.837686+0000","last_became_active":"2024-04-25T15:22:20.249792+0000","last_became_peered":"2024-04-25T15:22:20.249792+0000","last_unstale":"2024-04-25T15:22:21.837686+0000","last_undegraded":"2024-04-25T15:22:21.837686+0000","last_fullsized":"2024-04-25T15:22:21.837686+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:03.824723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245326+0000","last_change":"2024-04-25T15:22:20.241023+0000","last_active":"2024-04-25T15:22:21.245326+0000","last_peered":"2024-04-25T15:22:21.245326+0000","last_clean":"2024-04-25T15:22:21.245326+0000","last_became_active":"2024-04-25T15:22:20.240687+0000","last_became_peered":"2024-04-25T15:22:20.240687+0000","last_unstale":"2024-04-25T15:22:21.245326+0000","last_undegraded":"2024-04-25T15:22:21.245326+0000","last_fullsized":"2024-04-25T15:22:21.245326+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:02:40.755667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838577+0000","last_change":"2024-04-25T15:22:20.245544+0000","last_active":"2024-04-25T15:22:21.838577+0000","last_peered":"2024-04-25T15:22:21.838577+0000","last_clean":"2024-04-25T15:22:21.838577+0000","last_became_active":"2024-04-25T15:22:20.244783+0000","last_became_peered":"2024-04-25T15:22:20.244783+0000","last_unstale":"2024-04-25T15:22:21.838577+0000","last_undegraded":"2024-04-25T15:22:21.838577+0000","last_fullsized":"2024-04-25T15:22:21.838577+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:28:23.560582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838777+0000","last_change":"2024-04-25T15:22:20.252494+0000","last_active":"2024-04-25T15:22:21.838777+0000","last_peered":"2024-04-25T15:22:21.838777+0000","last_clean":"2024-04-25T15:22:21.838777+0000","last_became_active":"2024-04-25T15:22:20.251970+0000","last_became_peered":"2024-04-25T15:22:20.251970+0000","last_unstale":"2024-04-25T15:22:21.838777+0000","last_undegraded":"2024-04-25T15:22:21.838777+0000","last_fullsized":"2024-04-25T15:22:21.838777+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:53.128376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838111+0000","last_change":"2024-04-25T15:22:20.246159+0000","last_active":"2024-04-25T15:22:21.838111+0000","last_peered":"2024-04-25T15:22:21.838111+0000","last_clean":"2024-04-25T15:22:21.838111+0000","last_became_active":"2024-04-25T15:22:20.245829+0000","last_became_peered":"2024-04-25T15:22:20.245829+0000","last_unstale":"2024-04-25T15:22:21.838111+0000","last_undegraded":"2024-04-25T15:22:21.838111+0000","last_fullsized":"2024-04-25T15:22:21.838111+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:10.605540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838193+0000","last_change":"2024-04-25T15:22:20.251134+0000","last_active":"2024-04-25T15:22:21.838193+0000","last_peered":"2024-04-25T15:22:21.838193+0000","last_clean":"2024-04-25T15:22:21.838193+0000","last_became_active":"2024-04-25T15:22:20.247459+0000","last_became_peered":"2024-04-25T15:22:20.247459+0000","last_unstale":"2024-04-25T15:22:21.838193+0000","last_undegraded":"2024-04-25T15:22:21.838193+0000","last_fullsized":"2024-04-25T15:22:21.838193+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:26:04.288059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838857+0000","last_change":"2024-04-25T15:22:20.251658+0000","last_active":"2024-04-25T15:22:21.838857+0000","last_peered":"2024-04-25T15:22:21.838857+0000","last_clean":"2024-04-25T15:22:21.838857+0000","last_became_active":"2024-04-25T15:22:20.251348+0000","last_became_peered":"2024-04-25T15:22:20.251348+0000","last_unstale":"2024-04-25T15:22:21.838857+0000","last_undegraded":"2024-04-25T15:22:21.838857+0000","last_fullsized":"2024-04-25T15:22:21.838857+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:45.592275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838051+0000","last_change":"2024-04-25T15:22:20.248082+0000","last_active":"2024-04-25T15:22:21.838051+0000","last_peered":"2024-04-25T15:22:21.838051+0000","last_clean":"2024-04-25T15:22:21.838051+0000","last_became_active":"2024-04-25T15:22:20.247792+0000","last_became_peered":"2024-04-25T15:22:20.247792+0000","last_unstale":"2024-04-25T15:22:21.838051+0000","last_undegraded":"2024-04-25T15:22:21.838051+0000","last_fullsized":"2024-04-25T15:22:21.838051+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:51.604871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245468+0000","last_change":"2024-04-25T15:22:20.241552+0000","last_active":"2024-04-25T15:22:21.245468+0000","last_peered":"2024-04-25T15:22:21.245468+0000","last_clean":"2024-04-25T15:22:21.245468+0000","last_became_active":"2024-04-25T15:22:20.241184+0000","last_became_peered":"2024-04-25T15:22:20.241184+0000","last_unstale":"2024-04-25T15:22:21.245468+0000","last_undegraded":"2024-04-25T15:22:21.245468+0000","last_fullsized":"2024-04-25T15:22:21.245468+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:52.869453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245443+0000","last_change":"2024-04-25T15:22:20.241068+0000","last_active":"2024-04-25T15:22:21.245443+0000","last_peered":"2024-04-25T15:22:21.245443+0000","last_clean":"2024-04-25T15:22:21.245443+0000","last_became_active":"2024-04-25T15:22:20.240734+0000","last_became_peered":"2024-04-25T15:22:20.240734+0000","last_unstale":"2024-04-25T15:22:21.245443+0000","last_undegraded":"2024-04-25T15:22:21.245443+0000","last_fullsized":"2024-04-25T15:22:21.245443+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:28.791175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837479+0000","last_change":"2024-04-25T15:22:20.246819+0000","last_active":"2024-04-25T15:22:21.837479+0000","last_peered":"2024-04-25T15:22:21.837479+0000","last_clean":"2024-04-25T15:22:21.837479+0000","last_became_active":"2024-04-25T15:22:20.246517+0000","last_became_peered":"2024-04-25T15:22:20.246517+0000","last_unstale":"2024-04-25T15:22:21.837479+0000","last_undegraded":"2024-04-25T15:22:21.837479+0000","last_fullsized":"2024-04-25T15:22:21.837479+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:40.018439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245858+0000","last_change":"2024-04-25T15:22:20.245335+0000","last_active":"2024-04-25T15:22:21.245858+0000","last_peered":"2024-04-25T15:22:21.245858+0000","last_clean":"2024-04-25T15:22:21.245858+0000","last_became_active":"2024-04-25T15:22:20.245030+0000","last_became_peered":"2024-04-25T15:22:20.245030+0000","last_unstale":"2024-04-25T15:22:21.245858+0000","last_undegraded":"2024-04-25T15:22:21.245858+0000","last_fullsized":"2024-04-25T15:22:21.245858+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:53.128376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245450+0000","last_change":"2024-04-25T15:22:20.238751+0000","last_active":"2024-04-25T15:22:21.245450+0000","last_peered":"2024-04-25T15:22:21.245450+0000","last_clean":"2024-04-25T15:22:21.245450+0000","last_became_active":"2024-04-25T15:22:20.238304+0000","last_became_peered":"2024-04-25T15:22:20.238304+0000","last_unstale":"2024-04-25T15:22:21.245450+0000","last_undegraded":"2024-04-25T15:22:21.245450+0000","last_fullsized":"2024-04-25T15:22:21.245450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:47.230324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245644+0000","last_change":"2024-04-25T15:22:20.248270+0000","last_active":"2024-04-25T15:22:21.245644+0000","last_peered":"2024-04-25T15:22:21.245644+0000","last_clean":"2024-04-25T15:22:21.245644+0000","last_became_active":"2024-04-25T15:22:20.247234+0000","last_became_peered":"2024-04-25T15:22:20.247234+0000","last_unstale":"2024-04-25T15:22:21.245644+0000","last_undegraded":"2024-04-25T15:22:21.245644+0000","last_fullsized":"2024-04-25T15:22:21.245644+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:17.153800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838413+0000","last_change":"2024-04-25T15:22:20.250085+0000","last_active":"2024-04-25T15:22:21.838413+0000","last_peered":"2024-04-25T15:22:21.838413+0000","last_clean":"2024-04-25T15:22:21.838413+0000","last_became_active":"2024-04-25T15:22:20.248587+0000","last_became_peered":"2024-04-25T15:22:20.248587+0000","last_unstale":"2024-04-25T15:22:21.838413+0000","last_undegraded":"2024-04-25T15:22:21.838413+0000","last_fullsized":"2024-04-25T15:22:21.838413+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:14.060456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838176+0000","last_change":"2024-04-25T15:22:20.245454+0000","last_active":"2024-04-25T15:22:21.838176+0000","last_peered":"2024-04-25T15:22:21.838176+0000","last_clean":"2024-04-25T15:22:21.838176+0000","last_became_active":"2024-04-25T15:22:20.245068+0000","last_became_peered":"2024-04-25T15:22:20.245068+0000","last_unstale":"2024-04-25T15:22:21.838176+0000","last_undegraded":"2024-04-25T15:22:21.838176+0000","last_fullsized":"2024-04-25T15:22:21.838176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:26.464277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837721+0000","last_change":"2024-04-25T15:22:20.248469+0000","last_active":"2024-04-25T15:22:21.837721+0000","last_peered":"2024-04-25T15:22:21.837721+0000","last_clean":"2024-04-25T15:22:21.837721+0000","last_became_active":"2024-04-25T15:22:20.248151+0000","last_became_peered":"2024-04-25T15:22:20.248151+0000","last_unstale":"2024-04-25T15:22:21.837721+0000","last_undegraded":"2024-04-25T15:22:21.837721+0000","last_fullsized":"2024-04-25T15:22:21.837721+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:38.728895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837844+0000","last_change":"2024-04-25T15:22:20.248270+0000","last_active":"2024-04-25T15:22:21.837844+0000","last_peered":"2024-04-25T15:22:21.837844+0000","last_clean":"2024-04-25T15:22:21.837844+0000","last_became_active":"2024-04-25T15:22:20.247960+0000","last_became_peered":"2024-04-25T15:22:20.247960+0000","last_unstale":"2024-04-25T15:22:21.837844+0000","last_undegraded":"2024-04-25T15:22:21.837844+0000","last_fullsized":"2024-04-25T15:22:21.837844+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:06.982804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245593+0000","last_change":"2024-04-25T15:22:20.245536+0000","last_active":"2024-04-25T15:22:21.245593+0000","last_peered":"2024-04-25T15:22:21.245593+0000","last_clean":"2024-04-25T15:22:21.245593+0000","last_became_active":"2024-04-25T15:22:20.245229+0000","last_became_peered":"2024-04-25T15:22:20.245229+0000","last_unstale":"2024-04-25T15:22:21.245593+0000","last_undegraded":"2024-04-25T15:22:21.245593+0000","last_fullsized":"2024-04-25T15:22:21.245593+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:20.256045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245944+0000","last_change":"2024-04-25T15:22:20.245023+0000","last_active":"2024-04-25T15:22:21.245944+0000","last_peered":"2024-04-25T15:22:21.245944+0000","last_clean":"2024-04-25T15:22:21.245944+0000","last_became_active":"2024-04-25T15:22:20.244727+0000","last_became_peered":"2024-04-25T15:22:20.244727+0000","last_unstale":"2024-04-25T15:22:21.245944+0000","last_undegraded":"2024-04-25T15:22:21.245944+0000","last_fullsized":"2024-04-25T15:22:21.245944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838251+0000","last_change":"2024-04-25T15:22:20.257559+0000","last_active":"2024-04-25T15:22:21.838251+0000","last_peered":"2024-04-25T15:22:21.838251+0000","last_clean":"2024-04-25T15:22:21.838251+0000","last_became_active":"2024-04-25T15:22:20.257217+0000","last_became_peered":"2024-04-25T15:22:20.257217+0000","last_unstale":"2024-04-25T15:22:21.838251+0000","last_undegraded":"2024-04-25T15:22:21.838251+0000","last_fullsized":"2024-04-25T15:22:21.838251+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:13.143871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838642+0000","last_change":"2024-04-25T15:22:20.259024+0000","last_active":"2024-04-25T15:22:21.838642+0000","last_peered":"2024-04-25T15:22:21.838642+0000","last_clean":"2024-04-25T15:22:21.838642+0000","last_became_active":"2024-04-25T15:22:20.258739+0000","last_became_peered":"2024-04-25T15:22:20.258739+0000","last_unstale":"2024-04-25T15:22:21.838642+0000","last_undegraded":"2024-04-25T15:22:21.838642+0000","last_fullsized":"2024-04-25T15:22:21.838642+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:56.474098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838479+0000","last_change":"2024-04-25T15:22:20.249808+0000","last_active":"2024-04-25T15:22:21.838479+0000","last_peered":"2024-04-25T15:22:21.838479+0000","last_clean":"2024-04-25T15:22:21.838479+0000","last_became_active":"2024-04-25T15:22:20.247950+0000","last_became_peered":"2024-04-25T15:22:20.247950+0000","last_unstale":"2024-04-25T15:22:21.838479+0000","last_undegraded":"2024-04-25T15:22:21.838479+0000","last_fullsized":"2024-04-25T15:22:21.838479+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:15.192726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245432+0000","last_change":"2024-04-25T15:22:20.246709+0000","last_active":"2024-04-25T15:22:21.245432+0000","last_peered":"2024-04-25T15:22:21.245432+0000","last_clean":"2024-04-25T15:22:21.245432+0000","last_became_active":"2024-04-25T15:22:20.246175+0000","last_became_peered":"2024-04-25T15:22:20.246175+0000","last_unstale":"2024-04-25T15:22:21.245432+0000","last_undegraded":"2024-04-25T15:22:21.245432+0000","last_fullsized":"2024-04-25T15:22:21.245432+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:39:33.729791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245067+0000","last_change":"2024-04-25T15:22:20.252563+0000","last_active":"2024-04-25T15:22:21.245067+0000","last_peered":"2024-04-25T15:22:21.245067+0000","last_clean":"2024-04-25T15:22:21.245067+0000","last_became_active":"2024-04-25T15:22:20.251998+0000","last_became_peered":"2024-04-25T15:22:20.251998+0000","last_unstale":"2024-04-25T15:22:21.245067+0000","last_undegraded":"2024-04-25T15:22:21.245067+0000","last_fullsized":"2024-04-25T15:22:21.245067+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:52.399706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245676+0000","last_change":"2024-04-25T15:22:20.234468+0000","last_active":"2024-04-25T15:22:21.245676+0000","last_peered":"2024-04-25T15:22:21.245676+0000","last_clean":"2024-04-25T15:22:21.245676+0000","last_became_active":"2024-04-25T15:22:20.234154+0000","last_became_peered":"2024-04-25T15:22:20.234154+0000","last_unstale":"2024-04-25T15:22:21.245676+0000","last_undegraded":"2024-04-25T15:22:21.245676+0000","last_fullsized":"2024-04-25T15:22:21.245676+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:15.113865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244798+0000","last_change":"2024-04-25T15:22:20.242840+0000","last_active":"2024-04-25T15:22:21.244798+0000","last_peered":"2024-04-25T15:22:21.244798+0000","last_clean":"2024-04-25T15:22:21.244798+0000","last_became_active":"2024-04-25T15:22:20.242542+0000","last_became_peered":"2024-04-25T15:22:20.242542+0000","last_unstale":"2024-04-25T15:22:21.244798+0000","last_undegraded":"2024-04-25T15:22:21.244798+0000","last_fullsized":"2024-04-25T15:22:21.244798+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:28.791175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244536+0000","last_change":"2024-04-25T15:22:20.237909+0000","last_active":"2024-04-25T15:22:21.244536+0000","last_peered":"2024-04-25T15:22:21.244536+0000","last_clean":"2024-04-25T15:22:21.244536+0000","last_became_active":"2024-04-25T15:22:20.237518+0000","last_became_peered":"2024-04-25T15:22:20.237518+0000","last_unstale":"2024-04-25T15:22:21.244536+0000","last_undegraded":"2024-04-25T15:22:21.244536+0000","last_fullsized":"2024-04-25T15:22:21.244536+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.272586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"53'4","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838173+0000","last_change":"2024-04-25T15:22:19.221855+0000","last_active":"2024-04-25T15:22:21.838173+0000","last_peered":"2024-04-25T15:22:21.838173+0000","last_clean":"2024-04-25T15:22:21.838173+0000","last_became_active":"2024-04-25T15:22:19.221528+0000","last_became_peered":"2024-04-25T15:22:19.221528+0000","last_unstale":"2024-04-25T15:22:21.838173+0000","last_undegraded":"2024-04-25T15:22:21.838173+0000","last_fullsized":"2024-04-25T15:22:21.838173+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:53.381076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244677+0000","last_change":"2024-04-25T15:22:20.241803+0000","last_active":"2024-04-25T15:22:21.244677+0000","last_peered":"2024-04-25T15:22:21.244677+0000","last_clean":"2024-04-25T15:22:21.244677+0000","last_became_active":"2024-04-25T15:22:20.241506+0000","last_became_peered":"2024-04-25T15:22:20.241506+0000","last_unstale":"2024-04-25T15:22:21.244677+0000","last_undegraded":"2024-04-25T15:22:21.244677+0000","last_fullsized":"2024-04-25T15:22:21.244677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:04.130563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838036+0000","last_change":"2024-04-25T15:22:19.226387+0000","last_active":"2024-04-25T15:22:21.838036+0000","last_peered":"2024-04-25T15:22:21.838036+0000","last_clean":"2024-04-25T15:22:21.838036+0000","last_became_active":"2024-04-25T15:22:19.225618+0000","last_became_peered":"2024-04-25T15:22:19.225618+0000","last_unstale":"2024-04-25T15:22:21.838036+0000","last_undegraded":"2024-04-25T15:22:21.838036+0000","last_fullsized":"2024-04-25T15:22:21.838036+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:10.630788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838718+0000","last_change":"2024-04-25T15:22:20.246331+0000","last_active":"2024-04-25T15:22:21.838718+0000","last_peered":"2024-04-25T15:22:21.838718+0000","last_clean":"2024-04-25T15:22:21.838718+0000","last_became_active":"2024-04-25T15:22:20.245869+0000","last_became_peered":"2024-04-25T15:22:20.245869+0000","last_unstale":"2024-04-25T15:22:21.838718+0000","last_undegraded":"2024-04-25T15:22:21.838718+0000","last_fullsized":"2024-04-25T15:22:21.838718+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:27.649352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838760+0000","last_change":"2024-04-25T15:22:19.222761+0000","last_active":"2024-04-25T15:22:21.838760+0000","last_peered":"2024-04-25T15:22:21.838760+0000","last_clean":"2024-04-25T15:22:21.838760+0000","last_became_active":"2024-04-25T15:22:19.222346+0000","last_became_peered":"2024-04-25T15:22:19.222346+0000","last_unstale":"2024-04-25T15:22:21.838760+0000","last_undegraded":"2024-04-25T15:22:21.838760+0000","last_fullsized":"2024-04-25T15:22:21.838760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:21.948451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837535+0000","last_change":"2024-04-25T15:22:20.257809+0000","last_active":"2024-04-25T15:22:21.837535+0000","last_peered":"2024-04-25T15:22:21.837535+0000","last_clean":"2024-04-25T15:22:21.837535+0000","last_became_active":"2024-04-25T15:22:20.257293+0000","last_became_peered":"2024-04-25T15:22:20.257293+0000","last_unstale":"2024-04-25T15:22:21.837535+0000","last_undegraded":"2024-04-25T15:22:21.837535+0000","last_fullsized":"2024-04-25T15:22:21.837535+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:17.153800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244459+0000","last_change":"2024-04-25T15:22:19.228732+0000","last_active":"2024-04-25T15:22:21.244459+0000","last_peered":"2024-04-25T15:22:21.244459+0000","last_clean":"2024-04-25T15:22:21.244459+0000","last_became_active":"2024-04-25T15:22:19.228356+0000","last_became_peered":"2024-04-25T15:22:19.228356+0000","last_unstale":"2024-04-25T15:22:21.244459+0000","last_undegraded":"2024-04-25T15:22:21.244459+0000","last_fullsized":"2024-04-25T15:22:21.244459+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.094077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838500+0000","last_change":"2024-04-25T15:22:20.247789+0000","last_active":"2024-04-25T15:22:21.838500+0000","last_peered":"2024-04-25T15:22:21.838500+0000","last_clean":"2024-04-25T15:22:21.838500+0000","last_became_active":"2024-04-25T15:22:20.247457+0000","last_became_peered":"2024-04-25T15:22:20.247457+0000","last_unstale":"2024-04-25T15:22:21.838500+0000","last_undegraded":"2024-04-25T15:22:21.838500+0000","last_fullsized":"2024-04-25T15:22:21.838500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:40.018439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245545+0000","last_change":"2024-04-25T15:22:19.216042+0000","last_active":"2024-04-25T15:22:21.245545+0000","last_peered":"2024-04-25T15:22:21.245545+0000","last_clean":"2024-04-25T15:22:21.245545+0000","last_became_active":"2024-04-25T15:22:19.215693+0000","last_became_peered":"2024-04-25T15:22:19.215693+0000","last_unstale":"2024-04-25T15:22:21.245545+0000","last_undegraded":"2024-04-25T15:22:21.245545+0000","last_fullsized":"2024-04-25T15:22:21.245545+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.094077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245041+0000","last_change":"2024-04-25T15:22:20.252372+0000","last_active":"2024-04-25T15:22:21.245041+0000","last_peered":"2024-04-25T15:22:21.245041+0000","last_clean":"2024-04-25T15:22:21.245041+0000","last_became_active":"2024-04-25T15:22:20.251910+0000","last_became_peered":"2024-04-25T15:22:20.251910+0000","last_unstale":"2024-04-25T15:22:21.245041+0000","last_undegraded":"2024-04-25T15:22:21.245041+0000","last_fullsized":"2024-04-25T15:22:21.245041+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:10.605540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837969+0000","last_change":"2024-04-25T15:22:19.221751+0000","last_active":"2024-04-25T15:22:21.837969+0000","last_peered":"2024-04-25T15:22:21.837969+0000","last_clean":"2024-04-25T15:22:21.837969+0000","last_became_active":"2024-04-25T15:22:19.221414+0000","last_became_peered":"2024-04-25T15:22:19.221414+0000","last_unstale":"2024-04-25T15:22:21.837969+0000","last_undegraded":"2024-04-25T15:22:21.837969+0000","last_fullsized":"2024-04-25T15:22:21.837969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:18:15.935149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838621+0000","last_change":"2024-04-25T15:22:20.251107+0000","last_active":"2024-04-25T15:22:21.838621+0000","last_peered":"2024-04-25T15:22:21.838621+0000","last_clean":"2024-04-25T15:22:21.838621+0000","last_became_active":"2024-04-25T15:22:20.249062+0000","last_became_peered":"2024-04-25T15:22:20.249062+0000","last_unstale":"2024-04-25T15:22:21.838621+0000","last_undegraded":"2024-04-25T15:22:21.838621+0000","last_fullsized":"2024-04-25T15:22:21.838621+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:40.014636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838290+0000","last_change":"2024-04-25T15:22:19.229122+0000","last_active":"2024-04-25T15:22:21.838290+0000","last_peered":"2024-04-25T15:22:21.838290+0000","last_clean":"2024-04-25T15:22:21.838290+0000","last_became_active":"2024-04-25T15:22:19.228692+0000","last_became_peered":"2024-04-25T15:22:19.228692+0000","last_unstale":"2024-04-25T15:22:21.838290+0000","last_undegraded":"2024-04-25T15:22:21.838290+0000","last_fullsized":"2024-04-25T15:22:21.838290+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:54.589176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244994+0000","last_change":"2024-04-25T15:22:20.248386+0000","last_active":"2024-04-25T15:22:21.244994+0000","last_peered":"2024-04-25T15:22:21.244994+0000","last_clean":"2024-04-25T15:22:21.244994+0000","last_became_active":"2024-04-25T15:22:20.248039+0000","last_became_peered":"2024-04-25T15:22:20.248039+0000","last_unstale":"2024-04-25T15:22:21.244994+0000","last_undegraded":"2024-04-25T15:22:21.244994+0000","last_fullsized":"2024-04-25T15:22:21.244994+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:51.604871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245280+0000","last_change":"2024-04-25T15:22:19.225586+0000","last_active":"2024-04-25T15:22:21.245280+0000","last_peered":"2024-04-25T15:22:21.245280+0000","last_clean":"2024-04-25T15:22:21.245280+0000","last_became_active":"2024-04-25T15:22:19.225270+0000","last_became_peered":"2024-04-25T15:22:19.225270+0000","last_unstale":"2024-04-25T15:22:21.245280+0000","last_undegraded":"2024-04-25T15:22:21.245280+0000","last_fullsized":"2024-04-25T15:22:21.245280+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:14.187778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837845+0000","last_change":"2024-04-25T15:22:20.252091+0000","last_active":"2024-04-25T15:22:21.837845+0000","last_peered":"2024-04-25T15:22:21.837845+0000","last_clean":"2024-04-25T15:22:21.837845+0000","last_became_active":"2024-04-25T15:22:20.251599+0000","last_became_peered":"2024-04-25T15:22:20.251599+0000","last_unstale":"2024-04-25T15:22:21.837845+0000","last_undegraded":"2024-04-25T15:22:21.837845+0000","last_fullsized":"2024-04-25T15:22:21.837845+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:27.649352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245148+0000","last_change":"2024-04-25T15:22:19.226659+0000","last_active":"2024-04-25T15:22:21.245148+0000","last_peered":"2024-04-25T15:22:21.245148+0000","last_clean":"2024-04-25T15:22:21.245148+0000","last_became_active":"2024-04-25T15:22:19.226297+0000","last_became_peered":"2024-04-25T15:22:19.226297+0000","last_unstale":"2024-04-25T15:22:21.245148+0000","last_undegraded":"2024-04-25T15:22:21.245148+0000","last_fullsized":"2024-04-25T15:22:21.245148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:25.459329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244129+0000","last_change":"2024-04-25T15:22:20.246152+0000","last_active":"2024-04-25T15:22:21.244129+0000","last_peered":"2024-04-25T15:22:21.244129+0000","last_clean":"2024-04-25T15:22:21.244129+0000","last_became_active":"2024-04-25T15:22:20.245393+0000","last_became_peered":"2024-04-25T15:22:20.245393+0000","last_unstale":"2024-04-25T15:22:21.244129+0000","last_undegraded":"2024-04-25T15:22:21.244129+0000","last_fullsized":"2024-04-25T15:22:21.244129+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.272586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838660+0000","last_change":"2024-04-25T15:22:19.220589+0000","last_active":"2024-04-25T15:22:21.838660+0000","last_peered":"2024-04-25T15:22:21.838660+0000","last_clean":"2024-04-25T15:22:21.838660+0000","last_became_active":"2024-04-25T15:22:19.220258+0000","last_became_peered":"2024-04-25T15:22:19.220258+0000","last_unstale":"2024-04-25T15:22:21.838660+0000","last_undegraded":"2024-04-25T15:22:21.838660+0000","last_fullsized":"2024-04-25T15:22:21.838660+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:06.596472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245524+0000","last_change":"2024-04-25T15:22:20.246602+0000","last_active":"2024-04-25T15:22:21.245524+0000","last_peered":"2024-04-25T15:22:21.245524+0000","last_clean":"2024-04-25T15:22:21.245524+0000","last_became_active":"2024-04-25T15:22:20.246295+0000","last_became_peered":"2024-04-25T15:22:20.246295+0000","last_unstale":"2024-04-25T15:22:21.245524+0000","last_undegraded":"2024-04-25T15:22:21.245524+0000","last_fullsized":"2024-04-25T15:22:21.245524+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:44.506431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"53'3","reported_seq":17,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245245+0000","last_change":"2024-04-25T15:22:19.225288+0000","last_active":"2024-04-25T15:22:21.245245+0000","last_peered":"2024-04-25T15:22:21.245245+0000","last_clean":"2024-04-25T15:22:21.245245+0000","last_became_active":"2024-04-25T15:22:19.224956+0000","last_became_peered":"2024-04-25T15:22:19.224956+0000","last_unstale":"2024-04-25T15:22:21.245245+0000","last_undegraded":"2024-04-25T15:22:21.245245+0000","last_fullsized":"2024-04-25T15:22:21.245245+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:14.187778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838277+0000","last_change":"2024-04-25T15:22:20.245572+0000","last_active":"2024-04-25T15:22:21.838277+0000","last_peered":"2024-04-25T15:22:21.838277+0000","last_clean":"2024-04-25T15:22:21.838277+0000","last_became_active":"2024-04-25T15:22:20.244778+0000","last_became_peered":"2024-04-25T15:22:20.244778+0000","last_unstale":"2024-04-25T15:22:21.838277+0000","last_undegraded":"2024-04-25T15:22:21.838277+0000","last_fullsized":"2024-04-25T15:22:21.838277+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:17.517391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"53'6","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838938+0000","last_change":"2024-04-25T15:22:19.234098+0000","last_active":"2024-04-25T15:22:21.838938+0000","last_peered":"2024-04-25T15:22:21.838938+0000","last_clean":"2024-04-25T15:22:21.838938+0000","last_became_active":"2024-04-25T15:22:19.233752+0000","last_became_peered":"2024-04-25T15:22:19.233752+0000","last_unstale":"2024-04-25T15:22:21.838938+0000","last_undegraded":"2024-04-25T15:22:21.838938+0000","last_fullsized":"2024-04-25T15:22:21.838938+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:03.125615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244332+0000","last_change":"2024-04-25T15:22:20.241084+0000","last_active":"2024-04-25T15:22:21.244332+0000","last_peered":"2024-04-25T15:22:21.244332+0000","last_clean":"2024-04-25T15:22:21.244332+0000","last_became_active":"2024-04-25T15:22:20.240798+0000","last_became_peered":"2024-04-25T15:22:20.240798+0000","last_unstale":"2024-04-25T15:22:21.244332+0000","last_undegraded":"2024-04-25T15:22:21.244332+0000","last_fullsized":"2024-04-25T15:22:21.244332+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:51.604871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245555+0000","last_change":"2024-04-25T15:22:19.227176+0000","last_active":"2024-04-25T15:22:21.245555+0000","last_peered":"2024-04-25T15:22:21.245555+0000","last_clean":"2024-04-25T15:22:21.245555+0000","last_became_active":"2024-04-25T15:22:19.226275+0000","last_became_peered":"2024-04-25T15:22:19.226275+0000","last_unstale":"2024-04-25T15:22:21.245555+0000","last_undegraded":"2024-04-25T15:22:21.245555+0000","last_fullsized":"2024-04-25T15:22:21.245555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:45.068431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245816+0000","last_change":"2024-04-25T15:22:20.251823+0000","last_active":"2024-04-25T15:22:21.245816+0000","last_peered":"2024-04-25T15:22:21.245816+0000","last_clean":"2024-04-25T15:22:21.245816+0000","last_became_active":"2024-04-25T15:22:20.251517+0000","last_became_peered":"2024-04-25T15:22:20.251517+0000","last_unstale":"2024-04-25T15:22:21.245816+0000","last_undegraded":"2024-04-25T15:22:21.245816+0000","last_fullsized":"2024-04-25T15:22:21.245816+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:55.625265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838588+0000","last_change":"2024-04-25T15:22:19.226544+0000","last_active":"2024-04-25T15:22:21.838588+0000","last_peered":"2024-04-25T15:22:21.838588+0000","last_clean":"2024-04-25T15:22:21.838588+0000","last_became_active":"2024-04-25T15:22:19.226227+0000","last_became_peered":"2024-04-25T15:22:19.226227+0000","last_unstale":"2024-04-25T15:22:21.838588+0000","last_undegraded":"2024-04-25T15:22:21.838588+0000","last_fullsized":"2024-04-25T15:22:21.838588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:08.953914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838430+0000","last_change":"2024-04-25T15:22:20.249359+0000","last_active":"2024-04-25T15:22:21.838430+0000","last_peered":"2024-04-25T15:22:21.838430+0000","last_clean":"2024-04-25T15:22:21.838430+0000","last_became_active":"2024-04-25T15:22:20.248518+0000","last_became_peered":"2024-04-25T15:22:20.248518+0000","last_unstale":"2024-04-25T15:22:21.838430+0000","last_undegraded":"2024-04-25T15:22:21.838430+0000","last_fullsized":"2024-04-25T15:22:21.838430+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:55.625265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245463+0000","last_change":"2024-04-25T15:22:19.223403+0000","last_active":"2024-04-25T15:22:21.245463+0000","last_peered":"2024-04-25T15:22:21.245463+0000","last_clean":"2024-04-25T15:22:21.245463+0000","last_became_active":"2024-04-25T15:22:19.222564+0000","last_became_peered":"2024-04-25T15:22:19.222564+0000","last_unstale":"2024-04-25T15:22:21.245463+0000","last_undegraded":"2024-04-25T15:22:21.245463+0000","last_fullsized":"2024-04-25T15:22:21.245463+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:13.055507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245849+0000","last_change":"2024-04-25T15:22:20.241714+0000","last_active":"2024-04-25T15:22:21.245849+0000","last_peered":"2024-04-25T15:22:21.245849+0000","last_clean":"2024-04-25T15:22:21.245849+0000","last_became_active":"2024-04-25T15:22:20.241336+0000","last_became_peered":"2024-04-25T15:22:20.241336+0000","last_unstale":"2024-04-25T15:22:21.245849+0000","last_undegraded":"2024-04-25T15:22:21.245849+0000","last_fullsized":"2024-04-25T15:22:21.245849+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:40.014636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"53'8","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838721+0000","last_change":"2024-04-25T15:22:19.224410+0000","last_active":"2024-04-25T15:22:21.838721+0000","last_peered":"2024-04-25T15:22:21.838721+0000","last_clean":"2024-04-25T15:22:21.838721+0000","last_became_active":"2024-04-25T15:22:19.223436+0000","last_became_peered":"2024-04-25T15:22:19.223436+0000","last_unstale":"2024-04-25T15:22:21.838721+0000","last_undegraded":"2024-04-25T15:22:21.838721+0000","last_fullsized":"2024-04-25T15:22:21.838721+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:51.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838777+0000","last_change":"2024-04-25T15:22:20.246777+0000","last_active":"2024-04-25T15:22:21.838777+0000","last_peered":"2024-04-25T15:22:21.838777+0000","last_clean":"2024-04-25T15:22:21.838777+0000","last_became_active":"2024-04-25T15:22:20.246370+0000","last_became_peered":"2024-04-25T15:22:20.246370+0000","last_unstale":"2024-04-25T15:22:21.838777+0000","last_undegraded":"2024-04-25T15:22:21.838777+0000","last_fullsized":"2024-04-25T15:22:21.838777+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.246000+0000","last_change":"2024-04-25T15:22:19.227250+0000","last_active":"2024-04-25T15:22:21.246000+0000","last_peered":"2024-04-25T15:22:21.246000+0000","last_clean":"2024-04-25T15:22:21.246000+0000","last_became_active":"2024-04-25T15:22:19.226906+0000","last_became_peered":"2024-04-25T15:22:19.226906+0000","last_unstale":"2024-04-25T15:22:21.246000+0000","last_undegraded":"2024-04-25T15:22:21.246000+0000","last_fullsized":"2024-04-25T15:22:21.246000+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:39.013490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837443+0000","last_change":"2024-04-25T15:22:20.258636+0000","last_active":"2024-04-25T15:22:21.837443+0000","last_peered":"2024-04-25T15:22:21.837443+0000","last_clean":"2024-04-25T15:22:21.837443+0000","last_became_active":"2024-04-25T15:22:20.258335+0000","last_became_peered":"2024-04-25T15:22:20.258335+0000","last_unstale":"2024-04-25T15:22:21.837443+0000","last_undegraded":"2024-04-25T15:22:21.837443+0000","last_fullsized":"2024-04-25T15:22:21.837443+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:17.517391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838770+0000","last_change":"2024-04-25T15:22:19.229964+0000","last_active":"2024-04-25T15:22:21.838770+0000","last_peered":"2024-04-25T15:22:21.838770+0000","last_clean":"2024-04-25T15:22:21.838770+0000","last_became_active":"2024-04-25T15:22:19.229654+0000","last_became_peered":"2024-04-25T15:22:19.229654+0000","last_unstale":"2024-04-25T15:22:21.838770+0000","last_undegraded":"2024-04-25T15:22:21.838770+0000","last_fullsized":"2024-04-25T15:22:21.838770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:12.142746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837954+0000","last_change":"2024-04-25T15:22:20.248030+0000","last_active":"2024-04-25T15:22:21.837954+0000","last_peered":"2024-04-25T15:22:21.837954+0000","last_clean":"2024-04-25T15:22:21.837954+0000","last_became_active":"2024-04-25T15:22:20.247068+0000","last_became_peered":"2024-04-25T15:22:20.247068+0000","last_unstale":"2024-04-25T15:22:21.837954+0000","last_undegraded":"2024-04-25T15:22:21.837954+0000","last_fullsized":"2024-04-25T15:22:21.837954+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:46.073380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838793+0000","last_change":"2024-04-25T15:22:19.225546+0000","last_active":"2024-04-25T15:22:21.838793+0000","last_peered":"2024-04-25T15:22:21.838793+0000","last_clean":"2024-04-25T15:22:21.838793+0000","last_became_active":"2024-04-25T15:22:19.225134+0000","last_became_peered":"2024-04-25T15:22:19.225134+0000","last_unstale":"2024-04-25T15:22:21.838793+0000","last_undegraded":"2024-04-25T15:22:21.838793+0000","last_fullsized":"2024-04-25T15:22:21.838793+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:20:43.354402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838293+0000","last_change":"2024-04-25T15:22:20.248457+0000","last_active":"2024-04-25T15:22:21.838293+0000","last_peered":"2024-04-25T15:22:21.838293+0000","last_clean":"2024-04-25T15:22:21.838293+0000","last_became_active":"2024-04-25T15:22:20.247519+0000","last_became_peered":"2024-04-25T15:22:20.247519+0000","last_unstale":"2024-04-25T15:22:21.838293+0000","last_undegraded":"2024-04-25T15:22:21.838293+0000","last_fullsized":"2024-04-25T15:22:21.838293+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:52.399706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838559+0000","last_change":"2024-04-25T15:22:19.230111+0000","last_active":"2024-04-25T15:22:21.838559+0000","last_peered":"2024-04-25T15:22:21.838559+0000","last_clean":"2024-04-25T15:22:21.838559+0000","last_became_active":"2024-04-25T15:22:19.229838+0000","last_became_peered":"2024-04-25T15:22:19.229838+0000","last_unstale":"2024-04-25T15:22:21.838559+0000","last_undegraded":"2024-04-25T15:22:21.838559+0000","last_fullsized":"2024-04-25T15:22:21.838559+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:24.752892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244447+0000","last_change":"2024-04-25T15:22:20.245894+0000","last_active":"2024-04-25T15:22:21.244447+0000","last_peered":"2024-04-25T15:22:21.244447+0000","last_clean":"2024-04-25T15:22:21.244447+0000","last_became_active":"2024-04-25T15:22:20.245590+0000","last_became_peered":"2024-04-25T15:22:20.245590+0000","last_unstale":"2024-04-25T15:22:21.244447+0000","last_undegraded":"2024-04-25T15:22:21.244447+0000","last_fullsized":"2024-04-25T15:22:21.244447+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:52.399706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838209+0000","last_change":"2024-04-25T15:22:19.222609+0000","last_active":"2024-04-25T15:22:21.838209+0000","last_peered":"2024-04-25T15:22:21.838209+0000","last_clean":"2024-04-25T15:22:21.838209+0000","last_became_active":"2024-04-25T15:22:19.221260+0000","last_became_peered":"2024-04-25T15:22:19.221260+0000","last_unstale":"2024-04-25T15:22:21.838209+0000","last_undegraded":"2024-04-25T15:22:21.838209+0000","last_fullsized":"2024-04-25T15:22:21.838209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:06.596472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838480+0000","last_change":"2024-04-25T15:22:20.242094+0000","last_active":"2024-04-25T15:22:21.838480+0000","last_peered":"2024-04-25T15:22:21.838480+0000","last_clean":"2024-04-25T15:22:21.838480+0000","last_became_active":"2024-04-25T15:22:20.241734+0000","last_became_peered":"2024-04-25T15:22:20.241734+0000","last_unstale":"2024-04-25T15:22:21.838480+0000","last_undegraded":"2024-04-25T15:22:21.838480+0000","last_fullsized":"2024-04-25T15:22:21.838480+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:36.099025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837829+0000","last_change":"2024-04-25T15:22:19.234323+0000","last_active":"2024-04-25T15:22:21.837829+0000","last_peered":"2024-04-25T15:22:21.837829+0000","last_clean":"2024-04-25T15:22:21.837829+0000","last_became_active":"2024-04-25T15:22:19.233925+0000","last_became_peered":"2024-04-25T15:22:19.233925+0000","last_unstale":"2024-04-25T15:22:21.837829+0000","last_undegraded":"2024-04-25T15:22:21.837829+0000","last_fullsized":"2024-04-25T15:22:21.837829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:50.599923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245825+0000","last_change":"2024-04-25T15:22:20.246386+0000","last_active":"2024-04-25T15:22:21.245825+0000","last_peered":"2024-04-25T15:22:21.245825+0000","last_clean":"2024-04-25T15:22:21.245825+0000","last_became_active":"2024-04-25T15:22:20.246069+0000","last_became_peered":"2024-04-25T15:22:20.246069+0000","last_unstale":"2024-04-25T15:22:21.245825+0000","last_undegraded":"2024-04-25T15:22:21.245825+0000","last_fullsized":"2024-04-25T15:22:21.245825+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:28:23.560582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245602+0000","last_change":"2024-04-25T15:22:19.232299+0000","last_active":"2024-04-25T15:22:21.245602+0000","last_peered":"2024-04-25T15:22:21.245602+0000","last_clean":"2024-04-25T15:22:21.245602+0000","last_became_active":"2024-04-25T15:22:19.231972+0000","last_became_peered":"2024-04-25T15:22:19.231972+0000","last_unstale":"2024-04-25T15:22:21.245602+0000","last_undegraded":"2024-04-25T15:22:21.245602+0000","last_fullsized":"2024-04-25T15:22:21.245602+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:16.512443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837861+0000","last_change":"2024-04-25T15:22:20.252844+0000","last_active":"2024-04-25T15:22:21.837861+0000","last_peered":"2024-04-25T15:22:21.837861+0000","last_clean":"2024-04-25T15:22:21.837861+0000","last_became_active":"2024-04-25T15:22:20.252301+0000","last_became_peered":"2024-04-25T15:22:20.252301+0000","last_unstale":"2024-04-25T15:22:21.837861+0000","last_undegraded":"2024-04-25T15:22:21.837861+0000","last_fullsized":"2024-04-25T15:22:21.837861+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.520097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838136+0000","last_change":"2024-04-25T15:22:19.223923+0000","last_active":"2024-04-25T15:22:21.838136+0000","last_peered":"2024-04-25T15:22:21.838136+0000","last_clean":"2024-04-25T15:22:21.838136+0000","last_became_active":"2024-04-25T15:22:19.223599+0000","last_became_peered":"2024-04-25T15:22:19.223599+0000","last_unstale":"2024-04-25T15:22:21.838136+0000","last_undegraded":"2024-04-25T15:22:21.838136+0000","last_fullsized":"2024-04-25T15:22:21.838136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:13.158455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244528+0000","last_change":"2024-04-25T15:22:20.244432+0000","last_active":"2024-04-25T15:22:21.244528+0000","last_peered":"2024-04-25T15:22:21.244528+0000","last_clean":"2024-04-25T15:22:21.244528+0000","last_became_active":"2024-04-25T15:22:20.244070+0000","last_became_peered":"2024-04-25T15:22:20.244070+0000","last_unstale":"2024-04-25T15:22:21.244528+0000","last_undegraded":"2024-04-25T15:22:21.244528+0000","last_fullsized":"2024-04-25T15:22:21.244528+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:56.474098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838219+0000","last_change":"2024-04-25T15:22:19.227473+0000","last_active":"2024-04-25T15:22:21.838219+0000","last_peered":"2024-04-25T15:22:21.838219+0000","last_clean":"2024-04-25T15:22:21.838219+0000","last_became_active":"2024-04-25T15:22:19.227076+0000","last_became_peered":"2024-04-25T15:22:19.227076+0000","last_unstale":"2024-04-25T15:22:21.838219+0000","last_undegraded":"2024-04-25T15:22:21.838219+0000","last_fullsized":"2024-04-25T15:22:21.838219+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:12.264073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837998+0000","last_change":"2024-04-25T15:22:20.249037+0000","last_active":"2024-04-25T15:22:21.837998+0000","last_peered":"2024-04-25T15:22:21.837998+0000","last_clean":"2024-04-25T15:22:21.837998+0000","last_became_active":"2024-04-25T15:22:20.245854+0000","last_became_peered":"2024-04-25T15:22:20.245854+0000","last_unstale":"2024-04-25T15:22:21.837998+0000","last_undegraded":"2024-04-25T15:22:21.837998+0000","last_fullsized":"2024-04-25T15:22:21.837998+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.272586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245385+0000","last_change":"2024-04-25T15:22:19.220313+0000","last_active":"2024-04-25T15:22:21.245385+0000","last_peered":"2024-04-25T15:22:21.245385+0000","last_clean":"2024-04-25T15:22:21.245385+0000","last_became_active":"2024-04-25T15:22:19.219985+0000","last_became_peered":"2024-04-25T15:22:19.219985+0000","last_unstale":"2024-04-25T15:22:21.245385+0000","last_undegraded":"2024-04-25T15:22:21.245385+0000","last_fullsized":"2024-04-25T15:22:21.245385+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.094077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838233+0000","last_change":"2024-04-25T15:22:20.251591+0000","last_active":"2024-04-25T15:22:21.838233+0000","last_peered":"2024-04-25T15:22:21.838233+0000","last_clean":"2024-04-25T15:22:21.838233+0000","last_became_active":"2024-04-25T15:22:20.251189+0000","last_became_peered":"2024-04-25T15:22:20.251189+0000","last_unstale":"2024-04-25T15:22:21.838233+0000","last_undegraded":"2024-04-25T15:22:21.838233+0000","last_fullsized":"2024-04-25T15:22:21.838233+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:15.113865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245648+0000","last_change":"2024-04-25T15:22:19.219083+0000","last_active":"2024-04-25T15:22:21.245648+0000","last_peered":"2024-04-25T15:22:21.245648+0000","last_clean":"2024-04-25T15:22:21.245648+0000","last_became_active":"2024-04-25T15:22:19.218276+0000","last_became_peered":"2024-04-25T15:22:19.218276+0000","last_unstale":"2024-04-25T15:22:21.245648+0000","last_undegraded":"2024-04-25T15:22:21.245648+0000","last_fullsized":"2024-04-25T15:22:21.245648+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:26.644403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245569+0000","last_change":"2024-04-25T15:22:20.248290+0000","last_active":"2024-04-25T15:22:21.245569+0000","last_peered":"2024-04-25T15:22:21.245569+0000","last_clean":"2024-04-25T15:22:21.245569+0000","last_became_active":"2024-04-25T15:22:20.247970+0000","last_became_peered":"2024-04-25T15:22:20.247970+0000","last_unstale":"2024-04-25T15:22:21.245569+0000","last_undegraded":"2024-04-25T15:22:21.245569+0000","last_fullsized":"2024-04-25T15:22:21.245569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:31.467253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838359+0000","last_change":"2024-04-25T15:22:19.225386+0000","last_active":"2024-04-25T15:22:21.838359+0000","last_peered":"2024-04-25T15:22:21.838359+0000","last_clean":"2024-04-25T15:22:21.838359+0000","last_became_active":"2024-04-25T15:22:19.223492+0000","last_became_peered":"2024-04-25T15:22:19.223492+0000","last_unstale":"2024-04-25T15:22:21.838359+0000","last_undegraded":"2024-04-25T15:22:21.838359+0000","last_fullsized":"2024-04-25T15:22:21.838359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:55.469149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244353+0000","last_change":"2024-04-25T15:22:20.242618+0000","last_active":"2024-04-25T15:22:21.244353+0000","last_peered":"2024-04-25T15:22:21.244353+0000","last_clean":"2024-04-25T15:22:21.244353+0000","last_became_active":"2024-04-25T15:22:20.242272+0000","last_became_peered":"2024-04-25T15:22:20.242272+0000","last_unstale":"2024-04-25T15:22:21.244353+0000","last_undegraded":"2024-04-25T15:22:21.244353+0000","last_fullsized":"2024-04-25T15:22:21.244353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:10.605540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837935+0000","last_change":"2024-04-25T15:22:19.224843+0000","last_active":"2024-04-25T15:22:21.837935+0000","last_peered":"2024-04-25T15:22:21.837935+0000","last_clean":"2024-04-25T15:22:21.837935+0000","last_became_active":"2024-04-25T15:22:19.223639+0000","last_became_peered":"2024-04-25T15:22:19.223639+0000","last_unstale":"2024-04-25T15:22:21.837935+0000","last_undegraded":"2024-04-25T15:22:21.837935+0000","last_fullsized":"2024-04-25T15:22:21.837935+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:09.600592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245663+0000","last_change":"2024-04-25T15:22:20.235188+0000","last_active":"2024-04-25T15:22:21.245663+0000","last_peered":"2024-04-25T15:22:21.245663+0000","last_clean":"2024-04-25T15:22:21.245663+0000","last_became_active":"2024-04-25T15:22:20.234903+0000","last_became_peered":"2024-04-25T15:22:20.234903+0000","last_unstale":"2024-04-25T15:22:21.245663+0000","last_undegraded":"2024-04-25T15:22:21.245663+0000","last_fullsized":"2024-04-25T15:22:21.245663+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:38.728895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837885+0000","last_change":"2024-04-25T15:22:19.228557+0000","last_active":"2024-04-25T15:22:21.837885+0000","last_peered":"2024-04-25T15:22:21.837885+0000","last_clean":"2024-04-25T15:22:21.837885+0000","last_became_active":"2024-04-25T15:22:19.228192+0000","last_became_peered":"2024-04-25T15:22:19.228192+0000","last_unstale":"2024-04-25T15:22:21.837885+0000","last_undegraded":"2024-04-25T15:22:21.837885+0000","last_fullsized":"2024-04-25T15:22:21.837885+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:21.948451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838847+0000","last_change":"2024-04-25T15:22:20.240969+0000","last_active":"2024-04-25T15:22:21.838847+0000","last_peered":"2024-04-25T15:22:21.838847+0000","last_clean":"2024-04-25T15:22:21.838847+0000","last_became_active":"2024-04-25T15:22:20.240677+0000","last_became_peered":"2024-04-25T15:22:20.240677+0000","last_unstale":"2024-04-25T15:22:21.838847+0000","last_undegraded":"2024-04-25T15:22:21.838847+0000","last_fullsized":"2024-04-25T15:22:21.838847+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:02:00.830419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244642+0000","last_change":"2024-04-25T15:22:19.231086+0000","last_active":"2024-04-25T15:22:21.244642+0000","last_peered":"2024-04-25T15:22:21.244642+0000","last_clean":"2024-04-25T15:22:21.244642+0000","last_became_active":"2024-04-25T15:22:19.230690+0000","last_became_peered":"2024-04-25T15:22:19.230690+0000","last_unstale":"2024-04-25T15:22:21.244642+0000","last_undegraded":"2024-04-25T15:22:21.244642+0000","last_fullsized":"2024-04-25T15:22:21.244642+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:26.644403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838058+0000","last_change":"2024-04-25T15:22:20.252350+0000","last_active":"2024-04-25T15:22:21.838058+0000","last_peered":"2024-04-25T15:22:21.838058+0000","last_clean":"2024-04-25T15:22:21.838058+0000","last_became_active":"2024-04-25T15:22:20.250482+0000","last_became_peered":"2024-04-25T15:22:20.250482+0000","last_unstale":"2024-04-25T15:22:21.838058+0000","last_undegraded":"2024-04-25T15:22:21.838058+0000","last_fullsized":"2024-04-25T15:22:21.838058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:51.850024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"54'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:22.238819+0000","last_change":"2024-04-25T15:22:19.232163+0000","last_active":"2024-04-25T15:22:22.238819+0000","last_peered":"2024-04-25T15:22:22.238819+0000","last_clean":"2024-04-25T15:22:22.238819+0000","last_became_active":"2024-04-25T15:22:19.231839+0000","last_became_peered":"2024-04-25T15:22:19.231839+0000","last_unstale":"2024-04-25T15:22:22.238819+0000","last_undegraded":"2024-04-25T15:22:22.238819+0000","last_fullsized":"2024-04-25T15:22:22.238819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:15.700818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":175,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244598+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:22:21.244598+0000","last_peered":"2024-04-25T15:22:21.244598+0000","last_clean":"2024-04-25T15:22:21.244598+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:22:21.244598+0000","last_undegraded":"2024-04-25T15:22:21.244598+0000","last_fullsized":"2024-04-25T15:22:21.244598+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1245184,"data_stored":880692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":290852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":290200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":290782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":291834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":291812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":290360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1245184,"data_stored":880624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:22:24.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:22:24.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.385+0000 7f70df89a640 1 Processor -- start 2024-04-25T15:22:24.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.385+0000 7f70df89a640 1 -- start start 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70df89a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f70d80cbc90 0x7f70d8153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70df89a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70d80d01a0 0x7f70d815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70df89a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d80cf420 con 0x7f70d80d0ac0 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70df89a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f70d81541e0 con 0x7f70d80d01a0 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70df89a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f70d80cfd10 con 0x7f70d80cbc90 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70d80d01a0 0x7f70d815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70d80d01a0 0x7f70d815dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33594/0 (socket says 172.21.15.146:33594) 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 -- 172.21.15.146:0/2584251148 learned_addr learned my addr 172.21.15.146:0/2584251148 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:24.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dd636640 1 --2- 172.21.15.146:0/2584251148 >> v2:172.21.15.177:3300/0 conn(0x7f70d80cbc90 0x7f70d8153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 -- 172.21.15.146:0/2584251148 >> v2:172.21.15.177:3300/0 conn(0x7f70d80cbc90 msgr2=0x7f70d8153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:24.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 --2- 172.21.15.146:0/2584251148 >> v2:172.21.15.177:3300/0 conn(0x7f70d80cbc90 0x7f70d8153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 -- 172.21.15.146:0/2584251148 >> v1:172.21.15.177:6789/0 conn(0x7f70d80d0ac0 legacy=0x7f70d80d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:24.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.387+0000 7f70dce35640 1 -- 172.21.15.146:0/2584251148 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d80cc270 con 0x7f70d80d01a0 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.388+0000 7f70dce35640 1 --2- 172.21.15.146:0/2584251148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d815dc60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f70d80cfe90 tx=0x7f70c802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b48647e8e714a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.388+0000 7f70cffff640 1 -- 172.21.15.146:0/2584251148 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8037070 con 0x7f70d80d01a0 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.388+0000 7f70cffff640 1 -- 172.21.15.146:0/2584251148 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c8032440 con 0x7f70d80d01a0 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.388+0000 7f70cffff640 1 -- 172.21.15.146:0/2584251148 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8032810 con 0x7f70d80d01a0 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 -- 172.21.15.146:0/2584251148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 msgr2=0x7f70d815dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 --2- 172.21.15.146:0/2584251148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d815dc60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f70d80cfe90 tx=0x7f70c802a210 comp rx=0 tx=0).stop 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 -- 172.21.15.146:0/2584251148 shutdown_connections 2024-04-25T15:22:24.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 --2- 172.21.15.146:0/2584251148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d815dc60 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 --2- 172.21.15.146:0/2584251148 >> v2:172.21.15.177:3300/0 conn(0x7f70d80cbc90 0x7f70d8153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 -- 172.21.15.146:0/2584251148 >> 172.21.15.146:0/2584251148 conn(0x7f70d80ce0c0 msgr2=0x7f70d80ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:24.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 -- 172.21.15.146:0/2584251148 shutdown_connections 2024-04-25T15:22:24.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.389+0000 7f70df89a640 1 -- 172.21.15.146:0/2584251148 wait complete. 2024-04-25T15:22:24.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.391+0000 7f70df89a640 1 Processor -- start 2024-04-25T15:22:24.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.391+0000 7f70df89a640 1 -- start start 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70df89a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70d80cbc90 0x7f70d815ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70df89a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d81f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70df89a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70d80d0ac0 0x7f70d81f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70df89a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f70d815e1a0 con 0x7f70d80d01a0 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70df89a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70d80cf9a0 con 0x7f70d80cbc90 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70df89a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f70d80d05b0 con 0x7f70d80d0ac0 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d81f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d81f7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33596/0 (socket says 172.21.15.146:33596) 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 -- 172.21.15.146:0/2123988093 learned_addr learned my addr 172.21.15.146:0/2123988093 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dd636640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70d80cbc90 0x7f70d815ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dde37640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70d80d0ac0 0x7f70d81f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 -- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70d80d0ac0 msgr2=0x7f70d81f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70d80d0ac0 0x7f70d81f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 -- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70d80cbc90 msgr2=0x7f70d815ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70d80cbc90 0x7f70d815ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 -- 172.21.15.146:0/2123988093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d80b5310 con 0x7f70d80d01a0 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.392+0000 7f70dce35640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d81f7990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f70c8000c00 tx=0x7f70c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.393+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8041020 con 0x7f70d80d01a0 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.393+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c80306c0 con 0x7f70d80d01a0 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.393+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d80b4a70 con 0x7f70d80d01a0 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.393+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8037040 con 0x7f70d80d01a0 2024-04-25T15:22:24.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.393+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d80b5030 con 0x7f70d80d01a0 2024-04-25T15:22:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.394+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d8068c30 con 0x7f70d80d01a0 2024-04-25T15:22:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.395+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f70c8044020 con 0x7f70d80d01a0 2024-04-25T15:22:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.395+0000 7f70cdffb640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70bc074540 0x7f70bc076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.395+0000 7f70dd636640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70bc074540 0x7f70bc076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.395+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f70c80b5880 con 0x7f70d80d01a0 2024-04-25T15:22:24.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.396+0000 7f70dd636640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70bc074540 0x7f70bc076a50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f70d806c600 tx=0x7f70d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.400+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70d8068c30 con 0x7f70d80d01a0 2024-04-25T15:22:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.570+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f70d81f4240 con 0x7f70bc074540 2024-04-25T15:22:24.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.576+0000 7f70cdffb640 1 -- 172.21.15.146:0/2123988093 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f70d81f4240 con 0x7f70bc074540 2024-04-25T15:22:24.580 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:24.581 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70bc074540 msgr2=0x7f70bc076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70bc074540 0x7f70bc076a50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f70d806c600 tx=0x7f70d4008040 comp rx=0 tx=0).stop 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 msgr2=0x7f70d81f7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d81f7990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f70c8000c00 tx=0x7f70c80046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 shutdown_connections 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f70bc074540 0x7f70bc076a50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f70d80d0ac0 0x7f70d81f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70d80d01a0 0x7f70d81f7990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 --2- 172.21.15.146:0/2123988093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70d80cbc90 0x7f70d815ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.579+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 >> 172.21.15.146:0/2123988093 conn(0x7f70d80ce0c0 msgr2=0x7f70d815ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:24.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.580+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 shutdown_connections 2024-04-25T15:22:24.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.580+0000 7f70df89a640 1 -- 172.21.15.146:0/2123988093 wait complete. 2024-04-25T15:22:24.592 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-25T15:22:24.020691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222448,"kb_used_data":6248,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752272,"statfs":{"total":773094113280,"available":772866326528,"internally_reserved":0,"allocated":6397952,"data_stored":3507156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.817111"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837679+0000","last_change":"2024-04-25T15:22:20.247164+0000","last_active":"2024-04-25T15:22:21.837679+0000","last_peered":"2024-04-25T15:22:21.837679+0000","last_clean":"2024-04-25T15:22:21.837679+0000","last_became_active":"2024-04-25T15:22:20.246108+0000","last_became_peered":"2024-04-25T15:22:20.246108+0000","last_unstale":"2024-04-25T15:22:21.837679+0000","last_undegraded":"2024-04-25T15:22:21.837679+0000","last_fullsized":"2024-04-25T15:22:21.837679+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:11.666514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245214+0000","last_change":"2024-04-25T15:22:20.237989+0000","last_active":"2024-04-25T15:22:21.245214+0000","last_peered":"2024-04-25T15:22:21.245214+0000","last_clean":"2024-04-25T15:22:21.245214+0000","last_became_active":"2024-04-25T15:22:20.237719+0000","last_became_peered":"2024-04-25T15:22:20.237719+0000","last_unstale":"2024-04-25T15:22:21.245214+0000","last_undegraded":"2024-04-25T15:22:21.245214+0000","last_fullsized":"2024-04-25T15:22:21.245214+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:56.474098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245742+0000","last_change":"2024-04-25T15:22:20.240447+0000","last_active":"2024-04-25T15:22:21.245742+0000","last_peered":"2024-04-25T15:22:21.245742+0000","last_clean":"2024-04-25T15:22:21.245742+0000","last_became_active":"2024-04-25T15:22:20.240100+0000","last_became_peered":"2024-04-25T15:22:20.240100+0000","last_unstale":"2024-04-25T15:22:21.245742+0000","last_undegraded":"2024-04-25T15:22:21.245742+0000","last_fullsized":"2024-04-25T15:22:21.245742+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:13.143871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245312+0000","last_change":"2024-04-25T15:22:20.252667+0000","last_active":"2024-04-25T15:22:21.245312+0000","last_peered":"2024-04-25T15:22:21.245312+0000","last_clean":"2024-04-25T15:22:21.245312+0000","last_became_active":"2024-04-25T15:22:20.252363+0000","last_became_peered":"2024-04-25T15:22:20.252363+0000","last_unstale":"2024-04-25T15:22:21.245312+0000","last_undegraded":"2024-04-25T15:22:21.245312+0000","last_fullsized":"2024-04-25T15:22:21.245312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:04.130563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838181+0000","last_change":"2024-04-25T15:22:20.245860+0000","last_active":"2024-04-25T15:22:21.838181+0000","last_peered":"2024-04-25T15:22:21.838181+0000","last_clean":"2024-04-25T15:22:21.838181+0000","last_became_active":"2024-04-25T15:22:20.245248+0000","last_became_peered":"2024-04-25T15:22:20.245248+0000","last_unstale":"2024-04-25T15:22:21.838181+0000","last_undegraded":"2024-04-25T15:22:21.838181+0000","last_fullsized":"2024-04-25T15:22:21.838181+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:16.705766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837686+0000","last_change":"2024-04-25T15:22:20.250097+0000","last_active":"2024-04-25T15:22:21.837686+0000","last_peered":"2024-04-25T15:22:21.837686+0000","last_clean":"2024-04-25T15:22:21.837686+0000","last_became_active":"2024-04-25T15:22:20.249792+0000","last_became_peered":"2024-04-25T15:22:20.249792+0000","last_unstale":"2024-04-25T15:22:21.837686+0000","last_undegraded":"2024-04-25T15:22:21.837686+0000","last_fullsized":"2024-04-25T15:22:21.837686+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:03.824723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245326+0000","last_change":"2024-04-25T15:22:20.241023+0000","last_active":"2024-04-25T15:22:21.245326+0000","last_peered":"2024-04-25T15:22:21.245326+0000","last_clean":"2024-04-25T15:22:21.245326+0000","last_became_active":"2024-04-25T15:22:20.240687+0000","last_became_peered":"2024-04-25T15:22:20.240687+0000","last_unstale":"2024-04-25T15:22:21.245326+0000","last_undegraded":"2024-04-25T15:22:21.245326+0000","last_fullsized":"2024-04-25T15:22:21.245326+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:02:40.755667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838577+0000","last_change":"2024-04-25T15:22:20.245544+0000","last_active":"2024-04-25T15:22:21.838577+0000","last_peered":"2024-04-25T15:22:21.838577+0000","last_clean":"2024-04-25T15:22:21.838577+0000","last_became_active":"2024-04-25T15:22:20.244783+0000","last_became_peered":"2024-04-25T15:22:20.244783+0000","last_unstale":"2024-04-25T15:22:21.838577+0000","last_undegraded":"2024-04-25T15:22:21.838577+0000","last_fullsized":"2024-04-25T15:22:21.838577+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:28:23.560582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838777+0000","last_change":"2024-04-25T15:22:20.252494+0000","last_active":"2024-04-25T15:22:21.838777+0000","last_peered":"2024-04-25T15:22:21.838777+0000","last_clean":"2024-04-25T15:22:21.838777+0000","last_became_active":"2024-04-25T15:22:20.251970+0000","last_became_peered":"2024-04-25T15:22:20.251970+0000","last_unstale":"2024-04-25T15:22:21.838777+0000","last_undegraded":"2024-04-25T15:22:21.838777+0000","last_fullsized":"2024-04-25T15:22:21.838777+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:53.128376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838111+0000","last_change":"2024-04-25T15:22:20.246159+0000","last_active":"2024-04-25T15:22:21.838111+0000","last_peered":"2024-04-25T15:22:21.838111+0000","last_clean":"2024-04-25T15:22:21.838111+0000","last_became_active":"2024-04-25T15:22:20.245829+0000","last_became_peered":"2024-04-25T15:22:20.245829+0000","last_unstale":"2024-04-25T15:22:21.838111+0000","last_undegraded":"2024-04-25T15:22:21.838111+0000","last_fullsized":"2024-04-25T15:22:21.838111+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:10.605540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838193+0000","last_change":"2024-04-25T15:22:20.251134+0000","last_active":"2024-04-25T15:22:21.838193+0000","last_peered":"2024-04-25T15:22:21.838193+0000","last_clean":"2024-04-25T15:22:21.838193+0000","last_became_active":"2024-04-25T15:22:20.247459+0000","last_became_peered":"2024-04-25T15:22:20.247459+0000","last_unstale":"2024-04-25T15:22:21.838193+0000","last_undegraded":"2024-04-25T15:22:21.838193+0000","last_fullsized":"2024-04-25T15:22:21.838193+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:26:04.288059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838857+0000","last_change":"2024-04-25T15:22:20.251658+0000","last_active":"2024-04-25T15:22:21.838857+0000","last_peered":"2024-04-25T15:22:21.838857+0000","last_clean":"2024-04-25T15:22:21.838857+0000","last_became_active":"2024-04-25T15:22:20.251348+0000","last_became_peered":"2024-04-25T15:22:20.251348+0000","last_unstale":"2024-04-25T15:22:21.838857+0000","last_undegraded":"2024-04-25T15:22:21.838857+0000","last_fullsized":"2024-04-25T15:22:21.838857+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:45.592275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838051+0000","last_change":"2024-04-25T15:22:20.248082+0000","last_active":"2024-04-25T15:22:21.838051+0000","last_peered":"2024-04-25T15:22:21.838051+0000","last_clean":"2024-04-25T15:22:21.838051+0000","last_became_active":"2024-04-25T15:22:20.247792+0000","last_became_peered":"2024-04-25T15:22:20.247792+0000","last_unstale":"2024-04-25T15:22:21.838051+0000","last_undegraded":"2024-04-25T15:22:21.838051+0000","last_fullsized":"2024-04-25T15:22:21.838051+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:51.604871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245468+0000","last_change":"2024-04-25T15:22:20.241552+0000","last_active":"2024-04-25T15:22:21.245468+0000","last_peered":"2024-04-25T15:22:21.245468+0000","last_clean":"2024-04-25T15:22:21.245468+0000","last_became_active":"2024-04-25T15:22:20.241184+0000","last_became_peered":"2024-04-25T15:22:20.241184+0000","last_unstale":"2024-04-25T15:22:21.245468+0000","last_undegraded":"2024-04-25T15:22:21.245468+0000","last_fullsized":"2024-04-25T15:22:21.245468+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:52.869453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245443+0000","last_change":"2024-04-25T15:22:20.241068+0000","last_active":"2024-04-25T15:22:21.245443+0000","last_peered":"2024-04-25T15:22:21.245443+0000","last_clean":"2024-04-25T15:22:21.245443+0000","last_became_active":"2024-04-25T15:22:20.240734+0000","last_became_peered":"2024-04-25T15:22:20.240734+0000","last_unstale":"2024-04-25T15:22:21.245443+0000","last_undegraded":"2024-04-25T15:22:21.245443+0000","last_fullsized":"2024-04-25T15:22:21.245443+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:28.791175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837479+0000","last_change":"2024-04-25T15:22:20.246819+0000","last_active":"2024-04-25T15:22:21.837479+0000","last_peered":"2024-04-25T15:22:21.837479+0000","last_clean":"2024-04-25T15:22:21.837479+0000","last_became_active":"2024-04-25T15:22:20.246517+0000","last_became_peered":"2024-04-25T15:22:20.246517+0000","last_unstale":"2024-04-25T15:22:21.837479+0000","last_undegraded":"2024-04-25T15:22:21.837479+0000","last_fullsized":"2024-04-25T15:22:21.837479+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:40.018439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245858+0000","last_change":"2024-04-25T15:22:20.245335+0000","last_active":"2024-04-25T15:22:21.245858+0000","last_peered":"2024-04-25T15:22:21.245858+0000","last_clean":"2024-04-25T15:22:21.245858+0000","last_became_active":"2024-04-25T15:22:20.245030+0000","last_became_peered":"2024-04-25T15:22:20.245030+0000","last_unstale":"2024-04-25T15:22:21.245858+0000","last_undegraded":"2024-04-25T15:22:21.245858+0000","last_fullsized":"2024-04-25T15:22:21.245858+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:53.128376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245450+0000","last_change":"2024-04-25T15:22:20.238751+0000","last_active":"2024-04-25T15:22:21.245450+0000","last_peered":"2024-04-25T15:22:21.245450+0000","last_clean":"2024-04-25T15:22:21.245450+0000","last_became_active":"2024-04-25T15:22:20.238304+0000","last_became_peered":"2024-04-25T15:22:20.238304+0000","last_unstale":"2024-04-25T15:22:21.245450+0000","last_undegraded":"2024-04-25T15:22:21.245450+0000","last_fullsized":"2024-04-25T15:22:21.245450+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:47.230324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245644+0000","last_change":"2024-04-25T15:22:20.248270+0000","last_active":"2024-04-25T15:22:21.245644+0000","last_peered":"2024-04-25T15:22:21.245644+0000","last_clean":"2024-04-25T15:22:21.245644+0000","last_became_active":"2024-04-25T15:22:20.247234+0000","last_became_peered":"2024-04-25T15:22:20.247234+0000","last_unstale":"2024-04-25T15:22:21.245644+0000","last_undegraded":"2024-04-25T15:22:21.245644+0000","last_fullsized":"2024-04-25T15:22:21.245644+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:17.153800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838413+0000","last_change":"2024-04-25T15:22:20.250085+0000","last_active":"2024-04-25T15:22:21.838413+0000","last_peered":"2024-04-25T15:22:21.838413+0000","last_clean":"2024-04-25T15:22:21.838413+0000","last_became_active":"2024-04-25T15:22:20.248587+0000","last_became_peered":"2024-04-25T15:22:20.248587+0000","last_unstale":"2024-04-25T15:22:21.838413+0000","last_undegraded":"2024-04-25T15:22:21.838413+0000","last_fullsized":"2024-04-25T15:22:21.838413+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:14.060456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838176+0000","last_change":"2024-04-25T15:22:20.245454+0000","last_active":"2024-04-25T15:22:21.838176+0000","last_peered":"2024-04-25T15:22:21.838176+0000","last_clean":"2024-04-25T15:22:21.838176+0000","last_became_active":"2024-04-25T15:22:20.245068+0000","last_became_peered":"2024-04-25T15:22:20.245068+0000","last_unstale":"2024-04-25T15:22:21.838176+0000","last_undegraded":"2024-04-25T15:22:21.838176+0000","last_fullsized":"2024-04-25T15:22:21.838176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:26.464277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837721+0000","last_change":"2024-04-25T15:22:20.248469+0000","last_active":"2024-04-25T15:22:21.837721+0000","last_peered":"2024-04-25T15:22:21.837721+0000","last_clean":"2024-04-25T15:22:21.837721+0000","last_became_active":"2024-04-25T15:22:20.248151+0000","last_became_peered":"2024-04-25T15:22:20.248151+0000","last_unstale":"2024-04-25T15:22:21.837721+0000","last_undegraded":"2024-04-25T15:22:21.837721+0000","last_fullsized":"2024-04-25T15:22:21.837721+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:38.728895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837844+0000","last_change":"2024-04-25T15:22:20.248270+0000","last_active":"2024-04-25T15:22:21.837844+0000","last_peered":"2024-04-25T15:22:21.837844+0000","last_clean":"2024-04-25T15:22:21.837844+0000","last_became_active":"2024-04-25T15:22:20.247960+0000","last_became_peered":"2024-04-25T15:22:20.247960+0000","last_unstale":"2024-04-25T15:22:21.837844+0000","last_undegraded":"2024-04-25T15:22:21.837844+0000","last_fullsized":"2024-04-25T15:22:21.837844+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:06.982804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245593+0000","last_change":"2024-04-25T15:22:20.245536+0000","last_active":"2024-04-25T15:22:21.245593+0000","last_peered":"2024-04-25T15:22:21.245593+0000","last_clean":"2024-04-25T15:22:21.245593+0000","last_became_active":"2024-04-25T15:22:20.245229+0000","last_became_peered":"2024-04-25T15:22:20.245229+0000","last_unstale":"2024-04-25T15:22:21.245593+0000","last_undegraded":"2024-04-25T15:22:21.245593+0000","last_fullsized":"2024-04-25T15:22:21.245593+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:20.256045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245944+0000","last_change":"2024-04-25T15:22:20.245023+0000","last_active":"2024-04-25T15:22:21.245944+0000","last_peered":"2024-04-25T15:22:21.245944+0000","last_clean":"2024-04-25T15:22:21.245944+0000","last_became_active":"2024-04-25T15:22:20.244727+0000","last_became_peered":"2024-04-25T15:22:20.244727+0000","last_unstale":"2024-04-25T15:22:21.245944+0000","last_undegraded":"2024-04-25T15:22:21.245944+0000","last_fullsized":"2024-04-25T15:22:21.245944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838251+0000","last_change":"2024-04-25T15:22:20.257559+0000","last_active":"2024-04-25T15:22:21.838251+0000","last_peered":"2024-04-25T15:22:21.838251+0000","last_clean":"2024-04-25T15:22:21.838251+0000","last_became_active":"2024-04-25T15:22:20.257217+0000","last_became_peered":"2024-04-25T15:22:20.257217+0000","last_unstale":"2024-04-25T15:22:21.838251+0000","last_undegraded":"2024-04-25T15:22:21.838251+0000","last_fullsized":"2024-04-25T15:22:21.838251+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:13.143871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838642+0000","last_change":"2024-04-25T15:22:20.259024+0000","last_active":"2024-04-25T15:22:21.838642+0000","last_peered":"2024-04-25T15:22:21.838642+0000","last_clean":"2024-04-25T15:22:21.838642+0000","last_became_active":"2024-04-25T15:22:20.258739+0000","last_became_peered":"2024-04-25T15:22:20.258739+0000","last_unstale":"2024-04-25T15:22:21.838642+0000","last_undegraded":"2024-04-25T15:22:21.838642+0000","last_fullsized":"2024-04-25T15:22:21.838642+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:56.474098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838479+0000","last_change":"2024-04-25T15:22:20.249808+0000","last_active":"2024-04-25T15:22:21.838479+0000","last_peered":"2024-04-25T15:22:21.838479+0000","last_clean":"2024-04-25T15:22:21.838479+0000","last_became_active":"2024-04-25T15:22:20.247950+0000","last_became_peered":"2024-04-25T15:22:20.247950+0000","last_unstale":"2024-04-25T15:22:21.838479+0000","last_undegraded":"2024-04-25T15:22:21.838479+0000","last_fullsized":"2024-04-25T15:22:21.838479+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:15.192726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245432+0000","last_change":"2024-04-25T15:22:20.246709+0000","last_active":"2024-04-25T15:22:21.245432+0000","last_peered":"2024-04-25T15:22:21.245432+0000","last_clean":"2024-04-25T15:22:21.245432+0000","last_became_active":"2024-04-25T15:22:20.246175+0000","last_became_peered":"2024-04-25T15:22:20.246175+0000","last_unstale":"2024-04-25T15:22:21.245432+0000","last_undegraded":"2024-04-25T15:22:21.245432+0000","last_fullsized":"2024-04-25T15:22:21.245432+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:39:33.729791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245067+0000","last_change":"2024-04-25T15:22:20.252563+0000","last_active":"2024-04-25T15:22:21.245067+0000","last_peered":"2024-04-25T15:22:21.245067+0000","last_clean":"2024-04-25T15:22:21.245067+0000","last_became_active":"2024-04-25T15:22:20.251998+0000","last_became_peered":"2024-04-25T15:22:20.251998+0000","last_unstale":"2024-04-25T15:22:21.245067+0000","last_undegraded":"2024-04-25T15:22:21.245067+0000","last_fullsized":"2024-04-25T15:22:21.245067+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:52.399706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245676+0000","last_change":"2024-04-25T15:22:20.234468+0000","last_active":"2024-04-25T15:22:21.245676+0000","last_peered":"2024-04-25T15:22:21.245676+0000","last_clean":"2024-04-25T15:22:21.245676+0000","last_became_active":"2024-04-25T15:22:20.234154+0000","last_became_peered":"2024-04-25T15:22:20.234154+0000","last_unstale":"2024-04-25T15:22:21.245676+0000","last_undegraded":"2024-04-25T15:22:21.245676+0000","last_fullsized":"2024-04-25T15:22:21.245676+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:15.113865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244798+0000","last_change":"2024-04-25T15:22:20.242840+0000","last_active":"2024-04-25T15:22:21.244798+0000","last_peered":"2024-04-25T15:22:21.244798+0000","last_clean":"2024-04-25T15:22:21.244798+0000","last_became_active":"2024-04-25T15:22:20.242542+0000","last_became_peered":"2024-04-25T15:22:20.242542+0000","last_unstale":"2024-04-25T15:22:21.244798+0000","last_undegraded":"2024-04-25T15:22:21.244798+0000","last_fullsized":"2024-04-25T15:22:21.244798+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:28.791175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244536+0000","last_change":"2024-04-25T15:22:20.237909+0000","last_active":"2024-04-25T15:22:21.244536+0000","last_peered":"2024-04-25T15:22:21.244536+0000","last_clean":"2024-04-25T15:22:21.244536+0000","last_became_active":"2024-04-25T15:22:20.237518+0000","last_became_peered":"2024-04-25T15:22:20.237518+0000","last_unstale":"2024-04-25T15:22:21.244536+0000","last_undegraded":"2024-04-25T15:22:21.244536+0000","last_fullsized":"2024-04-25T15:22:21.244536+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.272586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"53'4","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838173+0000","last_change":"2024-04-25T15:22:19.221855+0000","last_active":"2024-04-25T15:22:21.838173+0000","last_peered":"2024-04-25T15:22:21.838173+0000","last_clean":"2024-04-25T15:22:21.838173+0000","last_became_active":"2024-04-25T15:22:19.221528+0000","last_became_peered":"2024-04-25T15:22:19.221528+0000","last_unstale":"2024-04-25T15:22:21.838173+0000","last_undegraded":"2024-04-25T15:22:21.838173+0000","last_fullsized":"2024-04-25T15:22:21.838173+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:53.381076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244677+0000","last_change":"2024-04-25T15:22:20.241803+0000","last_active":"2024-04-25T15:22:21.244677+0000","last_peered":"2024-04-25T15:22:21.244677+0000","last_clean":"2024-04-25T15:22:21.244677+0000","last_became_active":"2024-04-25T15:22:20.241506+0000","last_became_peered":"2024-04-25T15:22:20.241506+0000","last_unstale":"2024-04-25T15:22:21.244677+0000","last_undegraded":"2024-04-25T15:22:21.244677+0000","last_fullsized":"2024-04-25T15:22:21.244677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:04.130563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838036+0000","last_change":"2024-04-25T15:22:19.226387+0000","last_active":"2024-04-25T15:22:21.838036+0000","last_peered":"2024-04-25T15:22:21.838036+0000","last_clean":"2024-04-25T15:22:21.838036+0000","last_became_active":"2024-04-25T15:22:19.225618+0000","last_became_peered":"2024-04-25T15:22:19.225618+0000","last_unstale":"2024-04-25T15:22:21.838036+0000","last_undegraded":"2024-04-25T15:22:21.838036+0000","last_fullsized":"2024-04-25T15:22:21.838036+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:10.630788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838718+0000","last_change":"2024-04-25T15:22:20.246331+0000","last_active":"2024-04-25T15:22:21.838718+0000","last_peered":"2024-04-25T15:22:21.838718+0000","last_clean":"2024-04-25T15:22:21.838718+0000","last_became_active":"2024-04-25T15:22:20.245869+0000","last_became_peered":"2024-04-25T15:22:20.245869+0000","last_unstale":"2024-04-25T15:22:21.838718+0000","last_undegraded":"2024-04-25T15:22:21.838718+0000","last_fullsized":"2024-04-25T15:22:21.838718+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:27.649352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838760+0000","last_change":"2024-04-25T15:22:19.222761+0000","last_active":"2024-04-25T15:22:21.838760+0000","last_peered":"2024-04-25T15:22:21.838760+0000","last_clean":"2024-04-25T15:22:21.838760+0000","last_became_active":"2024-04-25T15:22:19.222346+0000","last_became_peered":"2024-04-25T15:22:19.222346+0000","last_unstale":"2024-04-25T15:22:21.838760+0000","last_undegraded":"2024-04-25T15:22:21.838760+0000","last_fullsized":"2024-04-25T15:22:21.838760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:21.948451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837535+0000","last_change":"2024-04-25T15:22:20.257809+0000","last_active":"2024-04-25T15:22:21.837535+0000","last_peered":"2024-04-25T15:22:21.837535+0000","last_clean":"2024-04-25T15:22:21.837535+0000","last_became_active":"2024-04-25T15:22:20.257293+0000","last_became_peered":"2024-04-25T15:22:20.257293+0000","last_unstale":"2024-04-25T15:22:21.837535+0000","last_undegraded":"2024-04-25T15:22:21.837535+0000","last_fullsized":"2024-04-25T15:22:21.837535+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:17.153800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244459+0000","last_change":"2024-04-25T15:22:19.228732+0000","last_active":"2024-04-25T15:22:21.244459+0000","last_peered":"2024-04-25T15:22:21.244459+0000","last_clean":"2024-04-25T15:22:21.244459+0000","last_became_active":"2024-04-25T15:22:19.228356+0000","last_became_peered":"2024-04-25T15:22:19.228356+0000","last_unstale":"2024-04-25T15:22:21.244459+0000","last_undegraded":"2024-04-25T15:22:21.244459+0000","last_fullsized":"2024-04-25T15:22:21.244459+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.094077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838500+0000","last_change":"2024-04-25T15:22:20.247789+0000","last_active":"2024-04-25T15:22:21.838500+0000","last_peered":"2024-04-25T15:22:21.838500+0000","last_clean":"2024-04-25T15:22:21.838500+0000","last_became_active":"2024-04-25T15:22:20.247457+0000","last_became_peered":"2024-04-25T15:22:20.247457+0000","last_unstale":"2024-04-25T15:22:21.838500+0000","last_undegraded":"2024-04-25T15:22:21.838500+0000","last_fullsized":"2024-04-25T15:22:21.838500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:40.018439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245545+0000","last_change":"2024-04-25T15:22:19.216042+0000","last_active":"2024-04-25T15:22:21.245545+0000","last_peered":"2024-04-25T15:22:21.245545+0000","last_clean":"2024-04-25T15:22:21.245545+0000","last_became_active":"2024-04-25T15:22:19.215693+0000","last_became_peered":"2024-04-25T15:22:19.215693+0000","last_unstale":"2024-04-25T15:22:21.245545+0000","last_undegraded":"2024-04-25T15:22:21.245545+0000","last_fullsized":"2024-04-25T15:22:21.245545+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.094077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245041+0000","last_change":"2024-04-25T15:22:20.252372+0000","last_active":"2024-04-25T15:22:21.245041+0000","last_peered":"2024-04-25T15:22:21.245041+0000","last_clean":"2024-04-25T15:22:21.245041+0000","last_became_active":"2024-04-25T15:22:20.251910+0000","last_became_peered":"2024-04-25T15:22:20.251910+0000","last_unstale":"2024-04-25T15:22:21.245041+0000","last_undegraded":"2024-04-25T15:22:21.245041+0000","last_fullsized":"2024-04-25T15:22:21.245041+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:10.605540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837969+0000","last_change":"2024-04-25T15:22:19.221751+0000","last_active":"2024-04-25T15:22:21.837969+0000","last_peered":"2024-04-25T15:22:21.837969+0000","last_clean":"2024-04-25T15:22:21.837969+0000","last_became_active":"2024-04-25T15:22:19.221414+0000","last_became_peered":"2024-04-25T15:22:19.221414+0000","last_unstale":"2024-04-25T15:22:21.837969+0000","last_undegraded":"2024-04-25T15:22:21.837969+0000","last_fullsized":"2024-04-25T15:22:21.837969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:18:15.935149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838621+0000","last_change":"2024-04-25T15:22:20.251107+0000","last_active":"2024-04-25T15:22:21.838621+0000","last_peered":"2024-04-25T15:22:21.838621+0000","last_clean":"2024-04-25T15:22:21.838621+0000","last_became_active":"2024-04-25T15:22:20.249062+0000","last_became_peered":"2024-04-25T15:22:20.249062+0000","last_unstale":"2024-04-25T15:22:21.838621+0000","last_undegraded":"2024-04-25T15:22:21.838621+0000","last_fullsized":"2024-04-25T15:22:21.838621+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:40.014636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838290+0000","last_change":"2024-04-25T15:22:19.229122+0000","last_active":"2024-04-25T15:22:21.838290+0000","last_peered":"2024-04-25T15:22:21.838290+0000","last_clean":"2024-04-25T15:22:21.838290+0000","last_became_active":"2024-04-25T15:22:19.228692+0000","last_became_peered":"2024-04-25T15:22:19.228692+0000","last_unstale":"2024-04-25T15:22:21.838290+0000","last_undegraded":"2024-04-25T15:22:21.838290+0000","last_fullsized":"2024-04-25T15:22:21.838290+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:54.589176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244994+0000","last_change":"2024-04-25T15:22:20.248386+0000","last_active":"2024-04-25T15:22:21.244994+0000","last_peered":"2024-04-25T15:22:21.244994+0000","last_clean":"2024-04-25T15:22:21.244994+0000","last_became_active":"2024-04-25T15:22:20.248039+0000","last_became_peered":"2024-04-25T15:22:20.248039+0000","last_unstale":"2024-04-25T15:22:21.244994+0000","last_undegraded":"2024-04-25T15:22:21.244994+0000","last_fullsized":"2024-04-25T15:22:21.244994+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:51.604871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245280+0000","last_change":"2024-04-25T15:22:19.225586+0000","last_active":"2024-04-25T15:22:21.245280+0000","last_peered":"2024-04-25T15:22:21.245280+0000","last_clean":"2024-04-25T15:22:21.245280+0000","last_became_active":"2024-04-25T15:22:19.225270+0000","last_became_peered":"2024-04-25T15:22:19.225270+0000","last_unstale":"2024-04-25T15:22:21.245280+0000","last_undegraded":"2024-04-25T15:22:21.245280+0000","last_fullsized":"2024-04-25T15:22:21.245280+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:14.187778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837845+0000","last_change":"2024-04-25T15:22:20.252091+0000","last_active":"2024-04-25T15:22:21.837845+0000","last_peered":"2024-04-25T15:22:21.837845+0000","last_clean":"2024-04-25T15:22:21.837845+0000","last_became_active":"2024-04-25T15:22:20.251599+0000","last_became_peered":"2024-04-25T15:22:20.251599+0000","last_unstale":"2024-04-25T15:22:21.837845+0000","last_undegraded":"2024-04-25T15:22:21.837845+0000","last_fullsized":"2024-04-25T15:22:21.837845+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:27.649352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245148+0000","last_change":"2024-04-25T15:22:19.226659+0000","last_active":"2024-04-25T15:22:21.245148+0000","last_peered":"2024-04-25T15:22:21.245148+0000","last_clean":"2024-04-25T15:22:21.245148+0000","last_became_active":"2024-04-25T15:22:19.226297+0000","last_became_peered":"2024-04-25T15:22:19.226297+0000","last_unstale":"2024-04-25T15:22:21.245148+0000","last_undegraded":"2024-04-25T15:22:21.245148+0000","last_fullsized":"2024-04-25T15:22:21.245148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:25.459329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244129+0000","last_change":"2024-04-25T15:22:20.246152+0000","last_active":"2024-04-25T15:22:21.244129+0000","last_peered":"2024-04-25T15:22:21.244129+0000","last_clean":"2024-04-25T15:22:21.244129+0000","last_became_active":"2024-04-25T15:22:20.245393+0000","last_became_peered":"2024-04-25T15:22:20.245393+0000","last_unstale":"2024-04-25T15:22:21.244129+0000","last_undegraded":"2024-04-25T15:22:21.244129+0000","last_fullsized":"2024-04-25T15:22:21.244129+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.272586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838660+0000","last_change":"2024-04-25T15:22:19.220589+0000","last_active":"2024-04-25T15:22:21.838660+0000","last_peered":"2024-04-25T15:22:21.838660+0000","last_clean":"2024-04-25T15:22:21.838660+0000","last_became_active":"2024-04-25T15:22:19.220258+0000","last_became_peered":"2024-04-25T15:22:19.220258+0000","last_unstale":"2024-04-25T15:22:21.838660+0000","last_undegraded":"2024-04-25T15:22:21.838660+0000","last_fullsized":"2024-04-25T15:22:21.838660+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:06.596472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245524+0000","last_change":"2024-04-25T15:22:20.246602+0000","last_active":"2024-04-25T15:22:21.245524+0000","last_peered":"2024-04-25T15:22:21.245524+0000","last_clean":"2024-04-25T15:22:21.245524+0000","last_became_active":"2024-04-25T15:22:20.246295+0000","last_became_peered":"2024-04-25T15:22:20.246295+0000","last_unstale":"2024-04-25T15:22:21.245524+0000","last_undegraded":"2024-04-25T15:22:21.245524+0000","last_fullsized":"2024-04-25T15:22:21.245524+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:44.506431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"53'3","reported_seq":17,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245245+0000","last_change":"2024-04-25T15:22:19.225288+0000","last_active":"2024-04-25T15:22:21.245245+0000","last_peered":"2024-04-25T15:22:21.245245+0000","last_clean":"2024-04-25T15:22:21.245245+0000","last_became_active":"2024-04-25T15:22:19.224956+0000","last_became_peered":"2024-04-25T15:22:19.224956+0000","last_unstale":"2024-04-25T15:22:21.245245+0000","last_undegraded":"2024-04-25T15:22:21.245245+0000","last_fullsized":"2024-04-25T15:22:21.245245+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:14.187778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838277+0000","last_change":"2024-04-25T15:22:20.245572+0000","last_active":"2024-04-25T15:22:21.838277+0000","last_peered":"2024-04-25T15:22:21.838277+0000","last_clean":"2024-04-25T15:22:21.838277+0000","last_became_active":"2024-04-25T15:22:20.244778+0000","last_became_peered":"2024-04-25T15:22:20.244778+0000","last_unstale":"2024-04-25T15:22:21.838277+0000","last_undegraded":"2024-04-25T15:22:21.838277+0000","last_fullsized":"2024-04-25T15:22:21.838277+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:17.517391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"53'6","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838938+0000","last_change":"2024-04-25T15:22:19.234098+0000","last_active":"2024-04-25T15:22:21.838938+0000","last_peered":"2024-04-25T15:22:21.838938+0000","last_clean":"2024-04-25T15:22:21.838938+0000","last_became_active":"2024-04-25T15:22:19.233752+0000","last_became_peered":"2024-04-25T15:22:19.233752+0000","last_unstale":"2024-04-25T15:22:21.838938+0000","last_undegraded":"2024-04-25T15:22:21.838938+0000","last_fullsized":"2024-04-25T15:22:21.838938+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:03.125615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244332+0000","last_change":"2024-04-25T15:22:20.241084+0000","last_active":"2024-04-25T15:22:21.244332+0000","last_peered":"2024-04-25T15:22:21.244332+0000","last_clean":"2024-04-25T15:22:21.244332+0000","last_became_active":"2024-04-25T15:22:20.240798+0000","last_became_peered":"2024-04-25T15:22:20.240798+0000","last_unstale":"2024-04-25T15:22:21.244332+0000","last_undegraded":"2024-04-25T15:22:21.244332+0000","last_fullsized":"2024-04-25T15:22:21.244332+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:51.604871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245555+0000","last_change":"2024-04-25T15:22:19.227176+0000","last_active":"2024-04-25T15:22:21.245555+0000","last_peered":"2024-04-25T15:22:21.245555+0000","last_clean":"2024-04-25T15:22:21.245555+0000","last_became_active":"2024-04-25T15:22:19.226275+0000","last_became_peered":"2024-04-25T15:22:19.226275+0000","last_unstale":"2024-04-25T15:22:21.245555+0000","last_undegraded":"2024-04-25T15:22:21.245555+0000","last_fullsized":"2024-04-25T15:22:21.245555+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:45.068431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245816+0000","last_change":"2024-04-25T15:22:20.251823+0000","last_active":"2024-04-25T15:22:21.245816+0000","last_peered":"2024-04-25T15:22:21.245816+0000","last_clean":"2024-04-25T15:22:21.245816+0000","last_became_active":"2024-04-25T15:22:20.251517+0000","last_became_peered":"2024-04-25T15:22:20.251517+0000","last_unstale":"2024-04-25T15:22:21.245816+0000","last_undegraded":"2024-04-25T15:22:21.245816+0000","last_fullsized":"2024-04-25T15:22:21.245816+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:55.625265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838588+0000","last_change":"2024-04-25T15:22:19.226544+0000","last_active":"2024-04-25T15:22:21.838588+0000","last_peered":"2024-04-25T15:22:21.838588+0000","last_clean":"2024-04-25T15:22:21.838588+0000","last_became_active":"2024-04-25T15:22:19.226227+0000","last_became_peered":"2024-04-25T15:22:19.226227+0000","last_unstale":"2024-04-25T15:22:21.838588+0000","last_undegraded":"2024-04-25T15:22:21.838588+0000","last_fullsized":"2024-04-25T15:22:21.838588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:08.953914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838430+0000","last_change":"2024-04-25T15:22:20.249359+0000","last_active":"2024-04-25T15:22:21.838430+0000","last_peered":"2024-04-25T15:22:21.838430+0000","last_clean":"2024-04-25T15:22:21.838430+0000","last_became_active":"2024-04-25T15:22:20.248518+0000","last_became_peered":"2024-04-25T15:22:20.248518+0000","last_unstale":"2024-04-25T15:22:21.838430+0000","last_undegraded":"2024-04-25T15:22:21.838430+0000","last_fullsized":"2024-04-25T15:22:21.838430+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:55.625265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245463+0000","last_change":"2024-04-25T15:22:19.223403+0000","last_active":"2024-04-25T15:22:21.245463+0000","last_peered":"2024-04-25T15:22:21.245463+0000","last_clean":"2024-04-25T15:22:21.245463+0000","last_became_active":"2024-04-25T15:22:19.222564+0000","last_became_peered":"2024-04-25T15:22:19.222564+0000","last_unstale":"2024-04-25T15:22:21.245463+0000","last_undegraded":"2024-04-25T15:22:21.245463+0000","last_fullsized":"2024-04-25T15:22:21.245463+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:13.055507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245849+0000","last_change":"2024-04-25T15:22:20.241714+0000","last_active":"2024-04-25T15:22:21.245849+0000","last_peered":"2024-04-25T15:22:21.245849+0000","last_clean":"2024-04-25T15:22:21.245849+0000","last_became_active":"2024-04-25T15:22:20.241336+0000","last_became_peered":"2024-04-25T15:22:20.241336+0000","last_unstale":"2024-04-25T15:22:21.245849+0000","last_undegraded":"2024-04-25T15:22:21.245849+0000","last_fullsized":"2024-04-25T15:22:21.245849+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:40.014636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"53'8","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838721+0000","last_change":"2024-04-25T15:22:19.224410+0000","last_active":"2024-04-25T15:22:21.838721+0000","last_peered":"2024-04-25T15:22:21.838721+0000","last_clean":"2024-04-25T15:22:21.838721+0000","last_became_active":"2024-04-25T15:22:19.223436+0000","last_became_peered":"2024-04-25T15:22:19.223436+0000","last_unstale":"2024-04-25T15:22:21.838721+0000","last_undegraded":"2024-04-25T15:22:21.838721+0000","last_fullsized":"2024-04-25T15:22:21.838721+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:51.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838777+0000","last_change":"2024-04-25T15:22:20.246777+0000","last_active":"2024-04-25T15:22:21.838777+0000","last_peered":"2024-04-25T15:22:21.838777+0000","last_clean":"2024-04-25T15:22:21.838777+0000","last_became_active":"2024-04-25T15:22:20.246370+0000","last_became_peered":"2024-04-25T15:22:20.246370+0000","last_unstale":"2024-04-25T15:22:21.838777+0000","last_undegraded":"2024-04-25T15:22:21.838777+0000","last_fullsized":"2024-04-25T15:22:21.838777+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.246000+0000","last_change":"2024-04-25T15:22:19.227250+0000","last_active":"2024-04-25T15:22:21.246000+0000","last_peered":"2024-04-25T15:22:21.246000+0000","last_clean":"2024-04-25T15:22:21.246000+0000","last_became_active":"2024-04-25T15:22:19.226906+0000","last_became_peered":"2024-04-25T15:22:19.226906+0000","last_unstale":"2024-04-25T15:22:21.246000+0000","last_undegraded":"2024-04-25T15:22:21.246000+0000","last_fullsized":"2024-04-25T15:22:21.246000+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:39.013490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837443+0000","last_change":"2024-04-25T15:22:20.258636+0000","last_active":"2024-04-25T15:22:21.837443+0000","last_peered":"2024-04-25T15:22:21.837443+0000","last_clean":"2024-04-25T15:22:21.837443+0000","last_became_active":"2024-04-25T15:22:20.258335+0000","last_became_peered":"2024-04-25T15:22:20.258335+0000","last_unstale":"2024-04-25T15:22:21.837443+0000","last_undegraded":"2024-04-25T15:22:21.837443+0000","last_fullsized":"2024-04-25T15:22:21.837443+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:17.517391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838770+0000","last_change":"2024-04-25T15:22:19.229964+0000","last_active":"2024-04-25T15:22:21.838770+0000","last_peered":"2024-04-25T15:22:21.838770+0000","last_clean":"2024-04-25T15:22:21.838770+0000","last_became_active":"2024-04-25T15:22:19.229654+0000","last_became_peered":"2024-04-25T15:22:19.229654+0000","last_unstale":"2024-04-25T15:22:21.838770+0000","last_undegraded":"2024-04-25T15:22:21.838770+0000","last_fullsized":"2024-04-25T15:22:21.838770+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:12.142746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837954+0000","last_change":"2024-04-25T15:22:20.248030+0000","last_active":"2024-04-25T15:22:21.837954+0000","last_peered":"2024-04-25T15:22:21.837954+0000","last_clean":"2024-04-25T15:22:21.837954+0000","last_became_active":"2024-04-25T15:22:20.247068+0000","last_became_peered":"2024-04-25T15:22:20.247068+0000","last_unstale":"2024-04-25T15:22:21.837954+0000","last_undegraded":"2024-04-25T15:22:21.837954+0000","last_fullsized":"2024-04-25T15:22:21.837954+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:46.073380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838793+0000","last_change":"2024-04-25T15:22:19.225546+0000","last_active":"2024-04-25T15:22:21.838793+0000","last_peered":"2024-04-25T15:22:21.838793+0000","last_clean":"2024-04-25T15:22:21.838793+0000","last_became_active":"2024-04-25T15:22:19.225134+0000","last_became_peered":"2024-04-25T15:22:19.225134+0000","last_unstale":"2024-04-25T15:22:21.838793+0000","last_undegraded":"2024-04-25T15:22:21.838793+0000","last_fullsized":"2024-04-25T15:22:21.838793+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:20:43.354402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838293+0000","last_change":"2024-04-25T15:22:20.248457+0000","last_active":"2024-04-25T15:22:21.838293+0000","last_peered":"2024-04-25T15:22:21.838293+0000","last_clean":"2024-04-25T15:22:21.838293+0000","last_became_active":"2024-04-25T15:22:20.247519+0000","last_became_peered":"2024-04-25T15:22:20.247519+0000","last_unstale":"2024-04-25T15:22:21.838293+0000","last_undegraded":"2024-04-25T15:22:21.838293+0000","last_fullsized":"2024-04-25T15:22:21.838293+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:52.399706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838559+0000","last_change":"2024-04-25T15:22:19.230111+0000","last_active":"2024-04-25T15:22:21.838559+0000","last_peered":"2024-04-25T15:22:21.838559+0000","last_clean":"2024-04-25T15:22:21.838559+0000","last_became_active":"2024-04-25T15:22:19.229838+0000","last_became_peered":"2024-04-25T15:22:19.229838+0000","last_unstale":"2024-04-25T15:22:21.838559+0000","last_undegraded":"2024-04-25T15:22:21.838559+0000","last_fullsized":"2024-04-25T15:22:21.838559+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:24.752892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244447+0000","last_change":"2024-04-25T15:22:20.245894+0000","last_active":"2024-04-25T15:22:21.244447+0000","last_peered":"2024-04-25T15:22:21.244447+0000","last_clean":"2024-04-25T15:22:21.244447+0000","last_became_active":"2024-04-25T15:22:20.245590+0000","last_became_peered":"2024-04-25T15:22:20.245590+0000","last_unstale":"2024-04-25T15:22:21.244447+0000","last_undegraded":"2024-04-25T15:22:21.244447+0000","last_fullsized":"2024-04-25T15:22:21.244447+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:52.399706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838209+0000","last_change":"2024-04-25T15:22:19.222609+0000","last_active":"2024-04-25T15:22:21.838209+0000","last_peered":"2024-04-25T15:22:21.838209+0000","last_clean":"2024-04-25T15:22:21.838209+0000","last_became_active":"2024-04-25T15:22:19.221260+0000","last_became_peered":"2024-04-25T15:22:19.221260+0000","last_unstale":"2024-04-25T15:22:21.838209+0000","last_undegraded":"2024-04-25T15:22:21.838209+0000","last_fullsized":"2024-04-25T15:22:21.838209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:06.596472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838480+0000","last_change":"2024-04-25T15:22:20.242094+0000","last_active":"2024-04-25T15:22:21.838480+0000","last_peered":"2024-04-25T15:22:21.838480+0000","last_clean":"2024-04-25T15:22:21.838480+0000","last_became_active":"2024-04-25T15:22:20.241734+0000","last_became_peered":"2024-04-25T15:22:20.241734+0000","last_unstale":"2024-04-25T15:22:21.838480+0000","last_undegraded":"2024-04-25T15:22:21.838480+0000","last_fullsized":"2024-04-25T15:22:21.838480+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:36.099025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837829+0000","last_change":"2024-04-25T15:22:19.234323+0000","last_active":"2024-04-25T15:22:21.837829+0000","last_peered":"2024-04-25T15:22:21.837829+0000","last_clean":"2024-04-25T15:22:21.837829+0000","last_became_active":"2024-04-25T15:22:19.233925+0000","last_became_peered":"2024-04-25T15:22:19.233925+0000","last_unstale":"2024-04-25T15:22:21.837829+0000","last_undegraded":"2024-04-25T15:22:21.837829+0000","last_fullsized":"2024-04-25T15:22:21.837829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:50.599923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245825+0000","last_change":"2024-04-25T15:22:20.246386+0000","last_active":"2024-04-25T15:22:21.245825+0000","last_peered":"2024-04-25T15:22:21.245825+0000","last_clean":"2024-04-25T15:22:21.245825+0000","last_became_active":"2024-04-25T15:22:20.246069+0000","last_became_peered":"2024-04-25T15:22:20.246069+0000","last_unstale":"2024-04-25T15:22:21.245825+0000","last_undegraded":"2024-04-25T15:22:21.245825+0000","last_fullsized":"2024-04-25T15:22:21.245825+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:28:23.560582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245602+0000","last_change":"2024-04-25T15:22:19.232299+0000","last_active":"2024-04-25T15:22:21.245602+0000","last_peered":"2024-04-25T15:22:21.245602+0000","last_clean":"2024-04-25T15:22:21.245602+0000","last_became_active":"2024-04-25T15:22:19.231972+0000","last_became_peered":"2024-04-25T15:22:19.231972+0000","last_unstale":"2024-04-25T15:22:21.245602+0000","last_undegraded":"2024-04-25T15:22:21.245602+0000","last_fullsized":"2024-04-25T15:22:21.245602+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:16.512443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837861+0000","last_change":"2024-04-25T15:22:20.252844+0000","last_active":"2024-04-25T15:22:21.837861+0000","last_peered":"2024-04-25T15:22:21.837861+0000","last_clean":"2024-04-25T15:22:21.837861+0000","last_became_active":"2024-04-25T15:22:20.252301+0000","last_became_peered":"2024-04-25T15:22:20.252301+0000","last_unstale":"2024-04-25T15:22:21.837861+0000","last_undegraded":"2024-04-25T15:22:21.837861+0000","last_fullsized":"2024-04-25T15:22:21.837861+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.520097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838136+0000","last_change":"2024-04-25T15:22:19.223923+0000","last_active":"2024-04-25T15:22:21.838136+0000","last_peered":"2024-04-25T15:22:21.838136+0000","last_clean":"2024-04-25T15:22:21.838136+0000","last_became_active":"2024-04-25T15:22:19.223599+0000","last_became_peered":"2024-04-25T15:22:19.223599+0000","last_unstale":"2024-04-25T15:22:21.838136+0000","last_undegraded":"2024-04-25T15:22:21.838136+0000","last_fullsized":"2024-04-25T15:22:21.838136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:13.158455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244528+0000","last_change":"2024-04-25T15:22:20.244432+0000","last_active":"2024-04-25T15:22:21.244528+0000","last_peered":"2024-04-25T15:22:21.244528+0000","last_clean":"2024-04-25T15:22:21.244528+0000","last_became_active":"2024-04-25T15:22:20.244070+0000","last_became_peered":"2024-04-25T15:22:20.244070+0000","last_unstale":"2024-04-25T15:22:21.244528+0000","last_undegraded":"2024-04-25T15:22:21.244528+0000","last_fullsized":"2024-04-25T15:22:21.244528+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:56.474098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838219+0000","last_change":"2024-04-25T15:22:19.227473+0000","last_active":"2024-04-25T15:22:21.838219+0000","last_peered":"2024-04-25T15:22:21.838219+0000","last_clean":"2024-04-25T15:22:21.838219+0000","last_became_active":"2024-04-25T15:22:19.227076+0000","last_became_peered":"2024-04-25T15:22:19.227076+0000","last_unstale":"2024-04-25T15:22:21.838219+0000","last_undegraded":"2024-04-25T15:22:21.838219+0000","last_fullsized":"2024-04-25T15:22:21.838219+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:12.264073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837998+0000","last_change":"2024-04-25T15:22:20.249037+0000","last_active":"2024-04-25T15:22:21.837998+0000","last_peered":"2024-04-25T15:22:21.837998+0000","last_clean":"2024-04-25T15:22:21.837998+0000","last_became_active":"2024-04-25T15:22:20.245854+0000","last_became_peered":"2024-04-25T15:22:20.245854+0000","last_unstale":"2024-04-25T15:22:21.837998+0000","last_undegraded":"2024-04-25T15:22:21.837998+0000","last_fullsized":"2024-04-25T15:22:21.837998+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.272586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245385+0000","last_change":"2024-04-25T15:22:19.220313+0000","last_active":"2024-04-25T15:22:21.245385+0000","last_peered":"2024-04-25T15:22:21.245385+0000","last_clean":"2024-04-25T15:22:21.245385+0000","last_became_active":"2024-04-25T15:22:19.219985+0000","last_became_peered":"2024-04-25T15:22:19.219985+0000","last_unstale":"2024-04-25T15:22:21.245385+0000","last_undegraded":"2024-04-25T15:22:21.245385+0000","last_fullsized":"2024-04-25T15:22:21.245385+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.094077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838233+0000","last_change":"2024-04-25T15:22:20.251591+0000","last_active":"2024-04-25T15:22:21.838233+0000","last_peered":"2024-04-25T15:22:21.838233+0000","last_clean":"2024-04-25T15:22:21.838233+0000","last_became_active":"2024-04-25T15:22:20.251189+0000","last_became_peered":"2024-04-25T15:22:20.251189+0000","last_unstale":"2024-04-25T15:22:21.838233+0000","last_undegraded":"2024-04-25T15:22:21.838233+0000","last_fullsized":"2024-04-25T15:22:21.838233+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:15.113865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245648+0000","last_change":"2024-04-25T15:22:19.219083+0000","last_active":"2024-04-25T15:22:21.245648+0000","last_peered":"2024-04-25T15:22:21.245648+0000","last_clean":"2024-04-25T15:22:21.245648+0000","last_became_active":"2024-04-25T15:22:19.218276+0000","last_became_peered":"2024-04-25T15:22:19.218276+0000","last_unstale":"2024-04-25T15:22:21.245648+0000","last_undegraded":"2024-04-25T15:22:21.245648+0000","last_fullsized":"2024-04-25T15:22:21.245648+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:26.644403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245569+0000","last_change":"2024-04-25T15:22:20.248290+0000","last_active":"2024-04-25T15:22:21.245569+0000","last_peered":"2024-04-25T15:22:21.245569+0000","last_clean":"2024-04-25T15:22:21.245569+0000","last_became_active":"2024-04-25T15:22:20.247970+0000","last_became_peered":"2024-04-25T15:22:20.247970+0000","last_unstale":"2024-04-25T15:22:21.245569+0000","last_undegraded":"2024-04-25T15:22:21.245569+0000","last_fullsized":"2024-04-25T15:22:21.245569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:31.467253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838359+0000","last_change":"2024-04-25T15:22:19.225386+0000","last_active":"2024-04-25T15:22:21.838359+0000","last_peered":"2024-04-25T15:22:21.838359+0000","last_clean":"2024-04-25T15:22:21.838359+0000","last_became_active":"2024-04-25T15:22:19.223492+0000","last_became_peered":"2024-04-25T15:22:19.223492+0000","last_unstale":"2024-04-25T15:22:21.838359+0000","last_undegraded":"2024-04-25T15:22:21.838359+0000","last_fullsized":"2024-04-25T15:22:21.838359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:55.469149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244353+0000","last_change":"2024-04-25T15:22:20.242618+0000","last_active":"2024-04-25T15:22:21.244353+0000","last_peered":"2024-04-25T15:22:21.244353+0000","last_clean":"2024-04-25T15:22:21.244353+0000","last_became_active":"2024-04-25T15:22:20.242272+0000","last_became_peered":"2024-04-25T15:22:20.242272+0000","last_unstale":"2024-04-25T15:22:21.244353+0000","last_undegraded":"2024-04-25T15:22:21.244353+0000","last_fullsized":"2024-04-25T15:22:21.244353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:10.605540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837935+0000","last_change":"2024-04-25T15:22:19.224843+0000","last_active":"2024-04-25T15:22:21.837935+0000","last_peered":"2024-04-25T15:22:21.837935+0000","last_clean":"2024-04-25T15:22:21.837935+0000","last_became_active":"2024-04-25T15:22:19.223639+0000","last_became_peered":"2024-04-25T15:22:19.223639+0000","last_unstale":"2024-04-25T15:22:21.837935+0000","last_undegraded":"2024-04-25T15:22:21.837935+0000","last_fullsized":"2024-04-25T15:22:21.837935+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:09.600592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.245663+0000","last_change":"2024-04-25T15:22:20.235188+0000","last_active":"2024-04-25T15:22:21.245663+0000","last_peered":"2024-04-25T15:22:21.245663+0000","last_clean":"2024-04-25T15:22:21.245663+0000","last_became_active":"2024-04-25T15:22:20.234903+0000","last_became_peered":"2024-04-25T15:22:20.234903+0000","last_unstale":"2024-04-25T15:22:21.245663+0000","last_undegraded":"2024-04-25T15:22:21.245663+0000","last_fullsized":"2024-04-25T15:22:21.245663+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:38.728895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.837885+0000","last_change":"2024-04-25T15:22:19.228557+0000","last_active":"2024-04-25T15:22:21.837885+0000","last_peered":"2024-04-25T15:22:21.837885+0000","last_clean":"2024-04-25T15:22:21.837885+0000","last_became_active":"2024-04-25T15:22:19.228192+0000","last_became_peered":"2024-04-25T15:22:19.228192+0000","last_unstale":"2024-04-25T15:22:21.837885+0000","last_undegraded":"2024-04-25T15:22:21.837885+0000","last_fullsized":"2024-04-25T15:22:21.837885+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:21.948451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838847+0000","last_change":"2024-04-25T15:22:20.240969+0000","last_active":"2024-04-25T15:22:21.838847+0000","last_peered":"2024-04-25T15:22:21.838847+0000","last_clean":"2024-04-25T15:22:21.838847+0000","last_became_active":"2024-04-25T15:22:20.240677+0000","last_became_peered":"2024-04-25T15:22:20.240677+0000","last_unstale":"2024-04-25T15:22:21.838847+0000","last_undegraded":"2024-04-25T15:22:21.838847+0000","last_fullsized":"2024-04-25T15:22:21.838847+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:02:00.830419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244642+0000","last_change":"2024-04-25T15:22:19.231086+0000","last_active":"2024-04-25T15:22:21.244642+0000","last_peered":"2024-04-25T15:22:21.244642+0000","last_clean":"2024-04-25T15:22:21.244642+0000","last_became_active":"2024-04-25T15:22:19.230690+0000","last_became_peered":"2024-04-25T15:22:19.230690+0000","last_unstale":"2024-04-25T15:22:21.244642+0000","last_undegraded":"2024-04-25T15:22:21.244642+0000","last_fullsized":"2024-04-25T15:22:21.244642+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:26.644403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.838058+0000","last_change":"2024-04-25T15:22:20.252350+0000","last_active":"2024-04-25T15:22:21.838058+0000","last_peered":"2024-04-25T15:22:21.838058+0000","last_clean":"2024-04-25T15:22:21.838058+0000","last_became_active":"2024-04-25T15:22:20.250482+0000","last_became_peered":"2024-04-25T15:22:20.250482+0000","last_unstale":"2024-04-25T15:22:21.838058+0000","last_undegraded":"2024-04-25T15:22:21.838058+0000","last_fullsized":"2024-04-25T15:22:21.838058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:19.202635+0000","last_clean_scrub_stamp":"2024-04-25T15:22:19.202635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:51.850024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"54'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:22.238819+0000","last_change":"2024-04-25T15:22:19.232163+0000","last_active":"2024-04-25T15:22:22.238819+0000","last_peered":"2024-04-25T15:22:22.238819+0000","last_clean":"2024-04-25T15:22:22.238819+0000","last_became_active":"2024-04-25T15:22:19.231839+0000","last_became_peered":"2024-04-25T15:22:19.231839+0000","last_unstale":"2024-04-25T15:22:22.238819+0000","last_undegraded":"2024-04-25T15:22:22.238819+0000","last_fullsized":"2024-04-25T15:22:22.238819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:22:18.197687+0000","last_clean_scrub_stamp":"2024-04-25T15:22:18.197687+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:15.700818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":175,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-25T15:22:21.244598+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:22:21.244598+0000","last_peered":"2024-04-25T15:22:21.244598+0000","last_clean":"2024-04-25T15:22:21.244598+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:22:21.244598+0000","last_undegraded":"2024-04-25T15:22:21.244598+0000","last_fullsized":"2024-04-25T15:22:21.244598+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1245184,"data_stored":880692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":290852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":290200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":290782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":291834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":291812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":290360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1245184,"data_stored":880624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:22:24.596 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:22:24.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:22:24.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.744+0000 7f06510ca640 1 Processor -- start 2024-04-25T15:22:24.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.744+0000 7f06510ca640 1 -- start start 2024-04-25T15:22:24.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.745+0000 7f06510ca640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f064c0d18e0 0x7f064c0ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.745+0000 7f06510ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c0cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.745+0000 7f06510ca640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f064c0cd720 0x7f064c0cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.745+0000 7f06510ca640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f064c0cd290 con 0x7f064c0cd720 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.745+0000 7f06510ca640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f064c0cfd70 con 0x7f064c0d18e0 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.745+0000 7f06510ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f064c070fd0 con 0x7f064c0cf420 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f064c0cd720 0x7f064c0cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f064c0cd720 0x7f064c0cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33610/0 (socket says 172.21.15.146:33610) 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 -- 172.21.15.146:0/2348676380 learned_addr learned my addr 172.21.15.146:0/2348676380 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064ad9d640 1 --2- 172.21.15.146:0/2348676380 >> v2:172.21.15.177:3300/0 conn(0x7f064c0d18e0 0x7f064c0ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064a59c640 1 --2- 172.21.15.146:0/2348676380 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c0cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 -- 172.21.15.146:0/2348676380 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 msgr2=0x7f064c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 --2- 172.21.15.146:0/2348676380 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c0cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 -- 172.21.15.146:0/2348676380 >> v2:172.21.15.177:3300/0 conn(0x7f064c0d18e0 msgr2=0x7f064c0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:24.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 --2- 172.21.15.146:0/2348676380 >> v2:172.21.15.177:3300/0 conn(0x7f064c0d18e0 0x7f064c0ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.746+0000 7f064b59e640 1 -- 172.21.15.146:0/2348676380 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064c0712b0 con 0x7f064c0cd720 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f064b59e640 1 --2- 172.21.15.146:0/2348676380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0cdb30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f064c0701f0 tx=0x7f064402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3b3734aa5f8bfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f0649d9b640 1 -- 172.21.15.146:0/2348676380 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644036070 con 0x7f064c0cd720 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f0649d9b640 1 -- 172.21.15.146:0/2348676380 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0644031600 con 0x7f064c0cd720 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f0649d9b640 1 -- 172.21.15.146:0/2348676380 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06440319d0 con 0x7f064c0cd720 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 -- 172.21.15.146:0/2348676380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 msgr2=0x7f064c0cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 --2- 172.21.15.146:0/2348676380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0cdb30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f064c0701f0 tx=0x7f064402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 -- 172.21.15.146:0/2348676380 shutdown_connections 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 --2- 172.21.15.146:0/2348676380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0cdb30 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 --2- 172.21.15.146:0/2348676380 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c0cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 --2- 172.21.15.146:0/2348676380 >> v2:172.21.15.177:3300/0 conn(0x7f064c0d18e0 0x7f064c0ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.747+0000 7f06510ca640 1 -- 172.21.15.146:0/2348676380 >> 172.21.15.146:0/2348676380 conn(0x7f064c09b2b0 msgr2=0x7f064c09b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:24.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.748+0000 7f06510ca640 1 -- 172.21.15.146:0/2348676380 shutdown_connections 2024-04-25T15:22:24.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.748+0000 7f06510ca640 1 -- 172.21.15.146:0/2348676380 wait complete. 2024-04-25T15:22:24.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.750+0000 7f06510ca640 1 Processor -- start 2024-04-25T15:22:24.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.750+0000 7f06510ca640 1 -- start start 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f06510ca640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f06510ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c161a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f06510ca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f064c0d18e0 0x7f064c161f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f06510ca640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f064c0d1cf0 con 0x7f064c0cd720 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f06510ca640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f064c0b1f00 con 0x7f064c0d18e0 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f06510ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f064c0d1e70 con 0x7f064c0cf420 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33612/0 (socket says 172.21.15.146:33612) 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 -- 172.21.15.146:0/2761043513 learned_addr learned my addr 172.21.15.146:0/2761043513 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064a59c640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064b59e640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f064c0d18e0 0x7f064c161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 -- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 msgr2=0x7f064c161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c161a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 -- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f064c0d18e0 msgr2=0x7f064c161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f064c0d18e0 0x7f064c161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.751+0000 7f064ad9d640 1 -- 172.21.15.146:0/2761043513 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064c0b44a0 con 0x7f064c0cd720 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.752+0000 7f064ad9d640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0b3f60 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f063c00e870 tx=0x7f063c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.752+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c017070 con 0x7f064c0cd720 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.752+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f063c00c940 con 0x7f064c0cd720 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.752+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064c162780 con 0x7f064c0cd720 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.752+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c00ccf0 con 0x7f064c0cd720 2024-04-25T15:22:24.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.752+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f064c162d40 con 0x7f064c0cd720 2024-04-25T15:22:24.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.753+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f064c0507b0 con 0x7f064c0cd720 2024-04-25T15:22:24.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.753+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f063c01a070 con 0x7f064c0cd720 2024-04-25T15:22:24.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.754+0000 7f06337fe640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0638074570 0x7f0638076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:24.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.754+0000 7f064a59c640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0638074570 0x7f0638076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:24.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.754+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f063c095750 con 0x7f064c0cd720 2024-04-25T15:22:24.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.755+0000 7f064a59c640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0638074570 0x7f0638076a80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f064c0d20b0 tx=0x7f0640009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:24.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:24.759+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f063c05f750 con 0x7f064c0cd720 2024-04-25T15:22:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.001+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f064c0b2080 con 0x7f064c0cd720 2024-04-25T15:22:25.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.004+0000 7f06337fe640 1 -- 172.21.15.146:0/2761043513 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7f063c063a50 con 0x7f064c0cd720 2024-04-25T15:22:25.006 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-04-25T15:22:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYdCpmlczBJhAApZj6jMADYxBHeBvhaprLDg== 2024-04-25T15:22:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: caps mds = "allow" 2024-04-25T15:22:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: caps mon = "allow r" 2024-04-25T15:22:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:22:25.007 INFO:teuthology.orchestra.run.smithi146.stderr:updated caps for client.0 2024-04-25T15:22:25.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0638074570 msgr2=0x7f0638076a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0638074570 0x7f0638076a80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f064c0d20b0 tx=0x7f0640009190 comp rx=0 tx=0).stop 2024-04-25T15:22:25.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 msgr2=0x7f064c0b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0b3f60 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f063c00e870 tx=0x7f063c00ed30 comp rx=0 tx=0).stop 2024-04-25T15:22:25.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 shutdown_connections 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0638074570 0x7f0638076a80 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f064c0d18e0 0x7f064c161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f064c0cf420 0x7f064c161a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 --2- 172.21.15.146:0/2761043513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f064c0cd720 0x7f064c0b3f60 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.006+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 >> 172.21.15.146:0/2761043513 conn(0x7f064c09b2b0 msgr2=0x7f064c0d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.007+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 shutdown_connections 2024-04-25T15:22:25.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.007+0000 7f06510ca640 1 -- 172.21.15.146:0/2761043513 wait complete. 2024-04-25T15:22:25.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:25.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.125+0000 7f81843d7640 1 Processor -- start 2024-04-25T15:22:25.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.125+0000 7f81843d7640 1 -- start start 2024-04-25T15:22:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f81843d7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0d12a0 0x7f817c15ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f81843d7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f817c0d0e10 0x7f817c06cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f81843d7640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c0d5000 con 0x7f817c0b45d0 2024-04-25T15:22:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f81843d7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f817c0d4db0 con 0x7f817c0d0e10 2024-04-25T15:22:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f81843d7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f817c0d52e0 con 0x7f817c0d12a0 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f817c0d0e10 0x7f817c06cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f817c0d0e10 0x7f817c06cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33620/0 (socket says 172.21.15.146:33620) 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 -- 172.21.15.146:0/1942301756 learned_addr learned my addr 172.21.15.146:0/1942301756 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182173640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0d12a0 0x7f817c15ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 -- 172.21.15.146:0/1942301756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0d12a0 msgr2=0x7f817c15ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0d12a0 0x7f817c15ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 -- 172.21.15.146:0/1942301756 >> v1:172.21.15.177:6789/0 conn(0x7f817c0b45d0 legacy=0x7f817c0d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.127+0000 7f8182974640 1 -- 172.21.15.146:0/1942301756 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c06c5f0 con 0x7f817c0d0e10 2024-04-25T15:22:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.128+0000 7f8182173640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0d12a0 0x7f817c15ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.128+0000 7f8182974640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0d0e10 0x7f817c06cdf0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8174009870 tx=0x7f817402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8383e9b486fab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.128+0000 7f8181171640 1 -- 172.21.15.146:0/1942301756 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174037070 con 0x7f817c0d0e10 2024-04-25T15:22:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.128+0000 7f8181171640 1 -- 172.21.15.146:0/1942301756 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817402ae10 con 0x7f817c0d0e10 2024-04-25T15:22:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.128+0000 7f8181171640 1 -- 172.21.15.146:0/1942301756 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174032660 con 0x7f817c0d0e10 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 -- 172.21.15.146:0/1942301756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0d0e10 msgr2=0x7f817c06cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0d0e10 0x7f817c06cdf0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f8174009870 tx=0x7f817402a100 comp rx=0 tx=0).stop 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 -- 172.21.15.146:0/1942301756 shutdown_connections 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0d0e10 0x7f817c06cdf0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 --2- 172.21.15.146:0/1942301756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0d12a0 0x7f817c15ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 -- 172.21.15.146:0/1942301756 >> 172.21.15.146:0/1942301756 conn(0x7f817c0d63b0 msgr2=0x7f817c0d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 -- 172.21.15.146:0/1942301756 shutdown_connections 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.129+0000 7f81843d7640 1 -- 172.21.15.146:0/1942301756 wait complete. 2024-04-25T15:22:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.131+0000 7f81843d7640 1 Processor -- start 2024-04-25T15:22:25.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.131+0000 7f81843d7640 1 -- start start 2024-04-25T15:22:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f81843d7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0b45d0 0x7f817c150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f81843d7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c0d12a0 0x7f817c0b9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f81843d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 0x7f817c15d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f81843d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0b9090 con 0x7f817c0b99c0 2024-04-25T15:22:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f81843d7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0b8f10 con 0x7f817c0d12a0 2024-04-25T15:22:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f81843d7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f817c0b4b20 con 0x7f817c0b45d0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f8182974640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 0x7f817c15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f8182974640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 0x7f817c15d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33622/0 (socket says 172.21.15.146:33622) 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f8182974640 1 -- 172.21.15.146:0/3358415969 learned_addr learned my addr 172.21.15.146:0/3358415969 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f8182173640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0b45d0 0x7f817c150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.132+0000 7f8181972640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c0d12a0 0x7f817c0b9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8182974640 1 -- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0b45d0 msgr2=0x7f817c150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8182974640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0b45d0 0x7f817c150730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8182974640 1 -- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c0d12a0 msgr2=0x7f817c0b9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8182974640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c0d12a0 0x7f817c0b9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8182974640 1 -- 172.21.15.146:0/3358415969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c14ece0 con 0x7f817c0b99c0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8182974640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 0x7f817c15d540 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f817402a140 tx=0x7f81740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174041070 con 0x7f817c0b99c0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817403c670 con 0x7f817c0b99c0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c14efc0 con 0x7f817c0b99c0 2024-04-25T15:22:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174037040 con 0x7f817c0b99c0 2024-04-25T15:22:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.133+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c14f580 con 0x7f817c0b99c0 2024-04-25T15:22:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.134+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817c14ea30 con 0x7f817c0b99c0 2024-04-25T15:22:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.135+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f8174044070 con 0x7f817c0b99c0 2024-04-25T15:22:25.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.135+0000 7f8172ffd640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8160074540 0x7f8160076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.135+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f81740b5a80 con 0x7f817c0b99c0 2024-04-25T15:22:25.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.135+0000 7f8182173640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8160074540 0x7f8160076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.136+0000 7f8182173640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8160074540 0x7f8160076a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f817c08def0 tx=0x7f81780077c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.140+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f817407fad0 con 0x7f817c0b99c0 2024-04-25T15:22:25.201 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-04-25T15:22:25.200+0000 7f930d81b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:22:25.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.352+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f817c0d3e30 con 0x7f817c0b99c0 2024-04-25T15:22:25.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.352+0000 7f8172ffd640 1 -- 172.21.15.146:0/3358415969 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5187 (secure 0 0 0) 0x7f8174083dd0 con 0x7f817c0b99c0 2024-04-25T15:22:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:25.354 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:22.605197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/865508848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":865508848},{"type":"v1","addr":"172.21.15.146:6836","nonce":865508848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":3}]} 2024-04-25T15:22:25.354 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 30 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8160074540 msgr2=0x7f8160076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8160074540 0x7f8160076a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f817c08def0 tx=0x7f81780077c0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 msgr2=0x7f817c15d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 0x7f817c15d540 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f817402a140 tx=0x7f81740046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 shutdown_connections 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8160074540 0x7f8160076a50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817c0b99c0 0x7f817c15d540 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f817c0d12a0 0x7f817c0b9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 --2- 172.21.15.146:0/3358415969 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f817c0b45d0 0x7f817c150730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 >> 172.21.15.146:0/3358415969 conn(0x7f817c0d63b0 msgr2=0x7f817c0ba1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 shutdown_connections 2024-04-25T15:22:25.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.354+0000 7f81843d7640 1 -- 172.21.15.146:0/3358415969 wait complete. 2024-04-25T15:22:25.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:22:19.583041+0000', 'modified': '2024-04-25T15:22:22.605197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6402}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6402': {'gid': 6402, 'name': 'c', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6836/865508848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 865508848}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 865508848}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6402, 'qdb_cluster': [6402]} 2024-04-25T15:22:25.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:22:25.771 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T15:22:25.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.881+0000 7fb9adb00640 1 Processor -- start 2024-04-25T15:22:25.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.881+0000 7fb9adb00640 1 -- start start 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9adb00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9a80b4df0 0x7fb9a8161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9adb00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9adb00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a8162520 con 0x7fb9a8162890 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9adb00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a80d23a0 con 0x7fb9a80b4df0 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9adb00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a80d1790 con 0x7fb9a8162110 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9a80b4df0 0x7fb9a8161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9a80b4df0 0x7fb9a8161bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33638/0 (socket says 172.21.15.146:33638) 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/1360339229 learned_addr learned my addr 172.21.15.146:0/1360339229 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/1360339229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/1360339229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 msgr2=0x7fb9a80ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/1360339229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a80ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/1360339229 >> v1:172.21.15.177:6789/0 conn(0x7fb9a8162890 legacy=0x7fb9a808d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.883+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/1360339229 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a80d1a70 con 0x7fb9a80b4df0 2024-04-25T15:22:25.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.884+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/1360339229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a8161bd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009870 tx=0x7fb99c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3260001db001d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.884+0000 7fb9a67fc640 1 -- 172.21.15.146:0/1360339229 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c037070 con 0x7fb9a80b4df0 2024-04-25T15:22:25.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.884+0000 7fb9a67fc640 1 -- 172.21.15.146:0/1360339229 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99c032440 con 0x7fb9a80b4df0 2024-04-25T15:22:25.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.884+0000 7fb9a67fc640 1 -- 172.21.15.146:0/1360339229 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c032810 con 0x7fb9a80b4df0 2024-04-25T15:22:25.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 -- 172.21.15.146:0/1360339229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 msgr2=0x7fb9a8161bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 --2- 172.21.15.146:0/1360339229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a8161bd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009870 tx=0x7fb99c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 -- 172.21.15.146:0/1360339229 shutdown_connections 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 --2- 172.21.15.146:0/1360339229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a80ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 --2- 172.21.15.146:0/1360339229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a8161bd0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 -- 172.21.15.146:0/1360339229 >> 172.21.15.146:0/1360339229 conn(0x7fb9a80b52e0 msgr2=0x7fb9a80b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 -- 172.21.15.146:0/1360339229 shutdown_connections 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.885+0000 7fb9adb00640 1 -- 172.21.15.146:0/1360339229 wait complete. 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.886+0000 7fb9adb00640 1 Processor -- start 2024-04-25T15:22:25.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.887+0000 7fb9adb00640 1 -- start start 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9adb00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9adb00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a809ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9adb00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9a8162890 0x7fb9a80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9adb00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a80d2830 con 0x7fb9a80b4df0 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9adb00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a80d2030 con 0x7fb9a8162890 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9adb00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a80ce940 con 0x7fb9a8162110 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33644/0 (socket says 172.21.15.146:33644) 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 -- 172.21.15.146:0/3824940706 learned_addr learned my addr 172.21.15.146:0/3824940706 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a809ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9a8162890 0x7fb9a80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 -- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 msgr2=0x7fb9a809ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a809ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 -- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9a8162890 msgr2=0x7fb9a80727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9a8162890 0x7fb9a80727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.888+0000 7fb9a77fe640 1 -- 172.21.15.146:0/3824940706 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a809dc40 con 0x7fb9a80b4df0 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.889+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a815b980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb99800c3c0 tx=0x7fb99800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.889+0000 7fb98ffff640 1 -- 172.21.15.146:0/3824940706 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9980052d0 con 0x7fb9a80b4df0 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.889+0000 7fb98ffff640 1 -- 172.21.15.146:0/3824940706 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9980054b0 con 0x7fb9a80b4df0 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.889+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a809df20 con 0x7fb9a80b4df0 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.889+0000 7fb98ffff640 1 -- 172.21.15.146:0/3824940706 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998006a60 con 0x7fb9a80b4df0 2024-04-25T15:22:25.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.889+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a809e4e0 con 0x7fb9a80b4df0 2024-04-25T15:22:25.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.890+0000 7fb98ffff640 1 -- 172.21.15.146:0/3824940706 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fb998006c00 con 0x7fb9a80b4df0 2024-04-25T15:22:25.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.891+0000 7fb98ffff640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb994074540 0x7fb994076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.891+0000 7fb98ffff640 1 -- 172.21.15.146:0/3824940706 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7fb998094820 con 0x7fb9a80b4df0 2024-04-25T15:22:25.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.891+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb994074540 0x7fb994076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.891+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb994074540 0x7fb994076a50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb99c034000 tx=0x7fb99c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.892+0000 7fb9b49b5740 1 Processor -- start 2024-04-25T15:22:25.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.892+0000 7fb9b49b5740 1 -- start start 2024-04-25T15:22:25.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023311600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102330dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610232e7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56102324b740 con 0x561022f24590 2024-04-25T15:22:25.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5610231e73a0 con 0x561022f45270 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x561023247700 con 0x561022f1c3b0 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610232e7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610232e7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33652/0 (socket says 172.21.15.146:33652) 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2884022502 learned_addr learned my addr 172.21.15.146:0/2884022502 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102330dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023311600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 msgr2=0x56102330dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102330dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 msgr2=0x561023311600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023311600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2884022502 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561023203750 con 0x561022f24590 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610232e7260 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009ed0 tx=0x7fb9a0007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171444800ae9a653 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb98d7fa640 1 -- 172.21.15.146:0/2884022502 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0013780 con 0x561022f24590 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb98d7fa640 1 -- 172.21.15.146:0/2884022502 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a0019070 con 0x561022f24590 2024-04-25T15:22:25.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.894+0000 7fb98d7fa640 1 -- 172.21.15.146:0/2884022502 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0013b20 con 0x561022f24590 2024-04-25T15:22:25.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2884022502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 msgr2=0x5610232e7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610232e7260 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009ed0 tx=0x7fb9a0007700 comp rx=0 tx=0).stop 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2884022502 shutdown_connections 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610232e7260 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102330dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2884022502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023311600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2884022502 >> 172.21.15.146:0/2884022502 conn(0x561022f393c0 msgr2=0x5610232e69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2884022502 shutdown_connections 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.895+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2884022502 wait complete. 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.896+0000 7fb9b49b5740 1 -- start start 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.896+0000 7fb9b49b5740 1 Processor -- start 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.897+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x561023314310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.897+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f24590 0x561023314850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.897+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 0x5610233093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5610231d2fe0 con 0x561022f45270 2024-04-25T15:22:25.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56102330f250 con 0x561022f24590 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x56102330f530 con 0x561022f1c3b0 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 10 client.0 _ll_register_callbacks cb 0x5610231e9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 0x5610233093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 0x5610233093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33666/0 (socket says 172.21.15.146:33666) 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2782498040 learned_addr learned my addr 172.21.15.146:0/2782498040 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x561023314310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f24590 0x561023314850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 msgr2=0x561023314310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x561023314310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f24590 msgr2=0x561023314850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f24590 0x561023314850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 -- 172.21.15.146:0/2782498040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561022f5d2d0 con 0x561022f45270 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f24590 0x561023314850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 0x5610233093c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0007c00 tx=0x7fb9a0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383b9cc442749b48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9867fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9867fc640 1 -- 172.21.15.146:0/2782498040 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0004240 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9867fc640 1 -- 172.21.15.146:0/2782498040 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7fb9a0029c70 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9a0019070 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9867fc640 1 -- 172.21.15.146:0/2782498040 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb9a0030050 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 10 client.6540 fetch_fsmap learned FSMap version 30 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.898+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5610230276b0 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9867fc640 1 -- 172.21.15.146:0/2782498040 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a0019070 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9867fc640 1 -- 172.21.15.146:0/2782498040 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fb9a0033020 con 0x561022f45270 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9b49b5740 10 client.6540 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9b49b5740 10 client.6540 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9b49b5740 20 client.6540 populate_metadata read hostname 'smithi146' 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561022f03e40 0x561023348ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9b49b5740 4 client.6540 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 --> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561023259350 con 0x561022f03e40 2024-04-25T15:22:25.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561022f03e40 0x561023348ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9867fc640 10 client.6540 ms_handle_connect on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.899+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561022f03e40 0x561023348ee0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ec4a4f30540aa1a server_cookie=644b7fdda042fda6 in_seq=0 out_seq=0 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9867fc640 1 -- 172.21.15.146:0/2782498040 <== mds.0 v2:172.21.15.146:6834/865508848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561023259350 con 0x561022f03e40 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9867fc640 10 client.6540 handle_command_reply: tid=0 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 client.6540 shutdown 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 msgr2=0x5610233093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 0x5610233093c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0007c00 tx=0x7fb9a0004ab0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 shutdown_connections 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561022f03e40 0x561023348ee0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f45270 0x5610233093c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f24590 0x561023314850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2782498040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x561023314310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9867fc640 0 client.6540 ms_handle_reset on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:25.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 >> 172.21.15.146:0/2782498040 conn(0x561022f393c0 msgr2=0x5610232e0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 shutdown_connections 2024-04-25T15:22:25.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2782498040 wait complete. 2024-04-25T15:22:25.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.900+0000 7fb9b49b5740 20 client.6540 trim_cache size 0 max 16384 2024-04-25T15:22:25.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.919+0000 7fb9b49b5740 1 Processor -- start 2024-04-25T15:22:25.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.919+0000 7fb9b49b5740 1 -- start start 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.920+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x5610233a7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.920+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x56102330c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.920+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023312330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.920+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a8200020 con 0x561022f24590 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.920+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a81e9400 con 0x561022f45270 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.920+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a81ffa30 con 0x561022f1c3b0 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x56102330c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x56102330c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33682/0 (socket says 172.21.15.146:33682) 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/3242781515 learned_addr learned my addr 172.21.15.146:0/3242781515 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x5610233a7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 msgr2=0x5610233a7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x5610233a7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 msgr2=0x561023312330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023312330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/3242781515 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561023313140 con 0x561022f24590 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x56102330c4e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb99c02a7b0 tx=0x7fb99c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6aefd647a4308d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb98cff9640 1 -- 172.21.15.146:0/3242781515 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c037040 con 0x561022f24590 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb98cff9640 1 -- 172.21.15.146:0/3242781515 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99c04f070 con 0x561022f24590 2024-04-25T15:22:25.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.921+0000 7fb98cff9640 1 -- 172.21.15.146:0/3242781515 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0044e0 con 0x561022f24590 2024-04-25T15:22:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 -- 172.21.15.146:0/3242781515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 msgr2=0x56102330c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x56102330c4e0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb99c02a7b0 tx=0x7fb99c02c000 comp rx=0 tx=0).stop 2024-04-25T15:22:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 -- 172.21.15.146:0/3242781515 shutdown_connections 2024-04-25T15:22:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x561023312330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x56102330c4e0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/3242781515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x5610233a7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 -- 172.21.15.146:0/3242781515 >> 172.21.15.146:0/3242781515 conn(0x561022f393c0 msgr2=0x5610233118e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 -- 172.21.15.146:0/3242781515 shutdown_connections 2024-04-25T15:22:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.922+0000 7fb9b49b5740 1 -- 172.21.15.146:0/3242781515 wait complete. 2024-04-25T15:22:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.923+0000 7fb9b49b5740 1 -- start start 2024-04-25T15:22:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.923+0000 7fb9b49b5740 1 Processor -- start 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102338e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610233a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x5610233a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561023344aa0 con 0x561022f24590 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561023310eb0 con 0x561022f45270 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x561023345030 con 0x561022f1c3b0 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9b49b5740 10 client.0 _ll_register_callbacks cb 0x7fb99c035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9a6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610233a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9a6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610233a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33686/0 (socket says 172.21.15.146:33686) 2024-04-25T15:22:25.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.924+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/2920551605 learned_addr learned my addr 172.21.15.146:0/2920551605 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102338e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x5610233a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 msgr2=0x56102338e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102338e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 msgr2=0x5610233a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x5610233a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a6ffd640 1 -- 172.21.15.146:0/2920551605 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610233143c0 con 0x561022f24590 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a7fff640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x5610233a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9a6ffd640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610233a59d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb99c000c00 tx=0x7fb99c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a632531dcb3c05c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9867fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9867fc640 1 -- 172.21.15.146:0/2920551605 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c057020 con 0x561022f24590 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9867fc640 1 -- 172.21.15.146:0/2920551605 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7fb99c04a460 con 0x561022f24590 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb99c04f070 con 0x561022f24590 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9867fc640 1 -- 172.21.15.146:0/2920551605 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb99c061050 con 0x561022f24590 2024-04-25T15:22:25.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 10 client.6546 fetch_fsmap learned FSMap version 30 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5610233a1c50 con 0x561022f24590 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9867fc640 1 -- 172.21.15.146:0/2920551605 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99c04f070 con 0x561022f24590 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9867fc640 1 -- 172.21.15.146:0/2920551605 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fb99c064020 con 0x561022f24590 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 10 client.6546 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 10 client.6546 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 20 client.6546 populate_metadata read hostname 'smithi146' 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561023343cd0 0x561023313310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 4 client.6546 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.925+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 --> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] -- command(tid 0: {"prefix": "status"}) -- 0x561023315670 con 0x561023343cd0 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561023343cd0 0x561023313310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9867fc640 10 client.6546 ms_handle_connect on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9a77fe640 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561023343cd0 0x561023313310 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52f65342080aaaa server_cookie=560f205d9df09266 in_seq=0 out_seq=0 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9867fc640 1 -- 172.21.15.146:0/2920551605 <== mds.0 v2:172.21.15.146:6834/865508848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x561023315670 con 0x561023343cd0 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9867fc640 10 client.6546 handle_command_reply: tid=0 2024-04-25T15:22:25.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 client.6546 shutdown 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 msgr2=0x5610233a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610233a59d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb99c000c00 tx=0x7fb99c004730 comp rx=0 tx=0).stop 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 shutdown_connections 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x561023343cd0 0x561023313310 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561022f45270 0x5610233a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561022f24590 0x5610233a59d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 --2- 172.21.15.146:0/2920551605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x561022f1c3b0 0x56102338e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9867fc640 0 client.6546 ms_handle_reset on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 >> 172.21.15.146:0/2920551605 conn(0x561022f393c0 msgr2=0x561023306fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 shutdown_connections 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.926+0000 7fb9b49b5740 1 -- 172.21.15.146:0/2920551605 wait complete. 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9b49b5740 20 client.6546 trim_cache size 0 max 16384 2024-04-25T15:22:25.933 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "c6e9a678-3722-48f4-a297-70f509c13e29", 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6402, 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 5.6945951829999997, 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 30, 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 54, 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 54, 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 11.655227030000001 2024-04-25T15:22:25.934 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:25.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb994074540 msgr2=0x7fb994076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb994074540 0x7fb994076a50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb99c034000 tx=0x7fb99c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:25.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 msgr2=0x7fb9a815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:25.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a815b980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb99800c3c0 tx=0x7fb99800c880 comp rx=0 tx=0).stop 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 shutdown_connections 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb994074540 0x7fb994076a50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9a8162890 0x7fb9a80727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9a8162110 0x7fb9a809ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 --2- 172.21.15.146:0/3824940706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a80b4df0 0x7fb9a815b980 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 >> 172.21.15.146:0/3824940706 conn(0x7fb9a80b52e0 msgr2=0x7fb9a8160f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 shutdown_connections 2024-04-25T15:22:25.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:25.927+0000 7fb9adb00640 1 -- 172.21.15.146:0/3824940706 wait complete. 2024-04-25T15:22:25.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:22:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.048+0000 7ff8558af640 1 Processor -- start 2024-04-25T15:22:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.048+0000 7ff8558af640 1 -- start start 2024-04-25T15:22:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff8558af640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8500b2df0 0x7ff8500d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff8558af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff850160680 con 0x7ff85006ee60 2024-04-25T15:22:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff8558af640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8500d20e0 con 0x7ff85008d370 2024-04-25T15:22:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff8558af640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff8500d23c0 con 0x7ff8500b2df0 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff84e7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff85006ee60 0x7ff85008cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50234/0 (socket says 172.21.15.146:50234) 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff84e7fc640 1 -- 172.21.15.146:0/1244552583 learned_addr learned my addr 172.21.15.146:0/1244552583 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff84effd640 1 --2- 172.21.15.146:0/1244552583 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8500b2df0 0x7ff8500d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff84dffb640 1 -- 172.21.15.146:0/1244552583 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818827582 0 0) 0x7ff850160680 con 0x7ff85006ee60 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.049+0000 7ff84dffb640 1 -- 172.21.15.146:0/1244552583 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff838003520 con 0x7ff85006ee60 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.050+0000 7ff84dffb640 1 -- 172.21.15.146:0/1244552583 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 932053188 0 0) 0x7ff838003520 con 0x7ff85006ee60 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.050+0000 7ff84effd640 1 -- 172.21.15.146:0/1244552583 >> v1:172.21.15.177:6789/0 conn(0x7ff85008d370 legacy=0x7ff8500cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.050+0000 7ff84effd640 1 -- 172.21.15.146:0/1244552583 >> v1:172.21.15.146:6789/0 conn(0x7ff85006ee60 legacy=0x7ff85008cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.050+0000 7ff84effd640 1 -- 172.21.15.146:0/1244552583 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8500d26a0 con 0x7ff8500b2df0 2024-04-25T15:22:26.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.050+0000 7ff84effd640 1 --2- 172.21.15.146:0/1244552583 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8500b2df0 0x7ff8500d3180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff840009e10 tx=0x7ff84002a540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4038b6131591a2f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:26.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.050+0000 7ff84dffb640 1 -- 172.21.15.146:0/1244552583 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840035020 con 0x7ff8500b2df0 2024-04-25T15:22:26.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff84dffb640 1 -- 172.21.15.146:0/1244552583 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff840007540 con 0x7ff8500b2df0 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff84dffb640 1 -- 172.21.15.146:0/1244552583 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff844001670 con 0x7ff8500b2df0 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 -- 172.21.15.146:0/1244552583 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8500b2df0 msgr2=0x7ff8500d3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 --2- 172.21.15.146:0/1244552583 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8500b2df0 0x7ff8500d3180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff840009e10 tx=0x7ff84002a540 comp rx=0 tx=0).stop 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 -- 172.21.15.146:0/1244552583 shutdown_connections 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 --2- 172.21.15.146:0/1244552583 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8500b2df0 0x7ff8500d3180 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 -- 172.21.15.146:0/1244552583 >> 172.21.15.146:0/1244552583 conn(0x7ff8500b5360 msgr2=0x7ff8500b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 -- 172.21.15.146:0/1244552583 shutdown_connections 2024-04-25T15:22:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.051+0000 7ff8558af640 1 -- 172.21.15.146:0/1244552583 wait complete. 2024-04-25T15:22:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.053+0000 7ff8558af640 1 Processor -- start 2024-04-25T15:22:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.053+0000 7ff8558af640 1 -- start start 2024-04-25T15:22:26.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.054+0000 7ff8558af640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff85006ee60 0x7ff8500c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:26.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.054+0000 7ff8558af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8500b2df0 0x7ff85014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:26.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.054+0000 7ff8558af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 0x7ff8500b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.054+0000 7ff8558af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff8500b91e0 con 0x7ff8500b4380 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.054+0000 7ff8558af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8500d2c60 con 0x7ff8500b2df0 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.054+0000 7ff8558af640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff8500d1c70 con 0x7ff85006ee60 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 0x7ff8500b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 0x7ff8500b4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33700/0 (socket says 172.21.15.146:33700) 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 -- 172.21.15.146:0/2527885855 learned_addr learned my addr 172.21.15.146:0/2527885855 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84e7fc640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8500b2df0 0x7ff85014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84effd640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff85006ee60 0x7ff8500c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 -- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff85006ee60 msgr2=0x7ff8500c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff85006ee60 0x7ff8500c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 -- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8500b2df0 msgr2=0x7ff85014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8500b2df0 0x7ff85014dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 -- 172.21.15.146:0/2527885855 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8500b4cd0 con 0x7ff8500b4380 2024-04-25T15:22:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff84f7fe640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 0x7ff8500b4790 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c4b0 tx=0x7ff84800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:26.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848017070 con 0x7ff8500b4380 2024-04-25T15:22:26.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff848005480 con 0x7ff8500b4380 2024-04-25T15:22:26.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8500c7d80 con 0x7ff8500b4380 2024-04-25T15:22:26.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848003830 con 0x7ff8500b4380 2024-04-25T15:22:26.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.055+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8500c8340 con 0x7ff8500b4380 2024-04-25T15:22:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.056+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85006a0e0 con 0x7ff8500b4380 2024-04-25T15:22:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.057+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7ff84801a070 con 0x7ff8500b4380 2024-04-25T15:22:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.058+0000 7ff837fff640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff83c074540 0x7ff83c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.058+0000 7ff84effd640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff83c074540 0x7ff83c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.058+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7ff848094620 con 0x7ff8500b4380 2024-04-25T15:22:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.058+0000 7ff84effd640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff83c074540 0x7ff83c076a50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff840033000 tx=0x7ff84002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:26.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.062+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8500b4cd0 con 0x7ff8500b4380 2024-04-25T15:22:26.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.278+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff8500b1c40 con 0x7ff8500b4380 2024-04-25T15:22:26.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.278+0000 7ff837fff640 1 -- 172.21.15.146:0/2527885855 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7ff84805e670 con 0x7ff8500b4380 2024-04-25T15:22:26.281 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.280+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff83c074540 msgr2=0x7ff83c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.280+0000 7ff8558af640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff83c074540 0x7ff83c076a50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff840033000 tx=0x7ff84002f000 comp rx=0 tx=0).stop 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.280+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 msgr2=0x7ff8500b4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.280+0000 7ff8558af640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 0x7ff8500b4790 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c4b0 tx=0x7ff84800c970 comp rx=0 tx=0).stop 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 shutdown_connections 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff83c074540 0x7ff83c076a50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8500b4380 0x7ff8500b4790 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8500b2df0 0x7ff85014dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 --2- 172.21.15.146:0/2527885855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff85006ee60 0x7ff8500c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:26.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 >> 172.21.15.146:0/2527885855 conn(0x7ff8500b5360 msgr2=0x7ff85015f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:26.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 shutdown_connections 2024-04-25T15:22:26.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26.281+0000 7ff8558af640 1 -- 172.21.15.146:0/2527885855 wait complete. 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:22:26.294 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:22:26.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:26.295 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:22:26.310 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:22:26.310 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:22:26.311 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42615sec preferred_lft 42615sec 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.312 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7820:1ff:feb4:c7f1/64 scope link 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:26.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:26.313 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:22:26.373 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:22:26.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:26.374 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:22:26.429 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:22:26.429 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:22:26.430 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:22:26.430 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:22:26.430 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:22:26.430 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:22:26.430 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:22:26.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:22:26.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:22:26.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:22:26.430 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:22:26.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:22:26.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:26.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:26.553 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:22:26.568 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:22:26.568 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:22:26.568 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:22:26.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:26.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-25T15:22:26.631 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T15:22:26.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:26.632 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:26.687 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:22:26.687 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:26.687 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:26.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:22:26.805 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:26.861 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:22:26.861 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:22:26.861 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:22:26.861 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:22:26.861 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:22:26.861 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:26.862 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:26.863 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:26.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:26.864 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:22:26.916 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:22:26.916 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-25T15:22:26.958 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:22:26.981 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:27.038 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:22:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:22:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:22:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:27.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.041 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:22:27.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:2024-04-25T15:22:27.047+0000 7fb567e31480 -1 init, newargv = 0x7fb54c00a5b0 newargc=15 2024-04-25T15:22:27.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[57653]: starting ceph client 2024-04-25T15:22:27.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[57653]: starting fuse 2024-04-25T15:22:27.063 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:22:27.064 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:22:27.064 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.0")) 2024-04-25T15:22:27.065 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:22:27.180 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.0.57653.asok 2024-04-25T15:22:27.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:27.190 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57653.asok 2024-04-25T15:22:27.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57653.asok status 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "0", 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "57653", 2024-04-25T15:22:27.335 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6555, 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 6555 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 510954298 2024-04-25T15:22:27.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 510954298 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.6555 192.168.0.1:0/510954298", 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/510954298", 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 30, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 54, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:22:27.337 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:27.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.344 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-04-25T15:22:27.344 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:27.359 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-04-25T15:22:27.359 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:27.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:27.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:22:27.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.571+0000 7f39aef2d640 1 Processor -- start 2024-04-25T15:22:27.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.571+0000 7f39aef2d640 1 -- start start 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.572+0000 7f39aef2d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 0x7f39a8168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.572+0000 7f39aef2d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a80cd490 con 0x7f39a81618f0 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.572+0000 7f39aef2d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a80cc880 con 0x7f39a80cc280 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.572+0000 7f39aef2d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f39a815bcd0 con 0x7f39a806d8d0 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.572+0000 7f39a7fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f39a81618f0 0x7f39a8161d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50248/0 (socket says 172.21.15.146:50248) 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.572+0000 7f39a7fff640 1 -- 172.21.15.146:0/2511315928 learned_addr learned my addr 172.21.15.146:0/2511315928 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39ad4ca640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 0x7f39a8168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641406353 0 0) 0x7f39a80cd490 con 0x7f39a81618f0 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3998003520 con 0x7f39a81618f0 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214477907 0 0) 0x7f39a80cc880 con 0x7f39a80cc280 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39a80cd490 con 0x7f39a80cc280 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3672043304 0 0) 0x7f3998003520 con 0x7f39a81618f0 2024-04-25T15:22:27.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 msgr2=0x7f39a8168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 0x7f39a8168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 >> v1:172.21.15.177:6789/0 conn(0x7f39a80cc280 legacy=0x7f39a8070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a80ccb60 con 0x7f39a81618f0 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f3994001670 con 0x7f39a81618f0 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 >> v1:172.21.15.146:6789/0 conn(0x7f39a81618f0 legacy=0x7f39a8161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3998003b70 0x7f3998003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 0x7f3998004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3998004d40 0x7f3998005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39a815bcd0 con 0x7f3998003b70 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f399800bb80 con 0x7f3998004d40 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f399800bd80 con 0x7f39a806d8d0 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39accc9640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3998003b70 0x7f3998003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39a7fff640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 0x7f3998004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39ad4ca640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3998004d40 0x7f3998005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39accc9640 1 -- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 msgr2=0x7f3998004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39accc9640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39a806d8d0 0x7f3998004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39accc9640 1 -- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3998004d40 msgr2=0x7f3998005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39accc9640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3998004d40 0x7f3998005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.573+0000 7f39accc9640 1 -- 172.21.15.146:0/2511315928 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3998003520 con 0x7f3998003b70 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.574+0000 7f39accc9640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3998003b70 0x7f3998003fc0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f399c0018d0 tx=0x7f3998005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c27fb9f198d794 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.574+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c0014f0 con 0x7f3998003b70 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.574+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f399c0052d0 con 0x7f3998003b70 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.574+0000 7f39a77fe640 1 -- 172.21.15.146:0/2511315928 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c02aca0 con 0x7f3998003b70 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39aef2d640 1 -- 172.21.15.146:0/2511315928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3998003b70 msgr2=0x7f3998003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39aef2d640 1 --2- 172.21.15.146:0/2511315928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3998003b70 0x7f3998003fc0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f399c0018d0 tx=0x7f3998005830 comp rx=0 tx=0).stop 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39ad4ca640 1 -- 172.21.15.146:0/2511315928 reap_dead start 2024-04-25T15:22:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39aef2d640 1 -- 172.21.15.146:0/2511315928 shutdown_connections 2024-04-25T15:22:27.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39aef2d640 1 -- 172.21.15.146:0/2511315928 >> 172.21.15.146:0/2511315928 conn(0x7f39a80ce0c0 msgr2=0x7f39a80ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:27.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39aef2d640 1 -- 172.21.15.146:0/2511315928 shutdown_connections 2024-04-25T15:22:27.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.575+0000 7f39aef2d640 1 -- 172.21.15.146:0/2511315928 wait complete. 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.577+0000 7f39aef2d640 1 Processor -- start 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.577+0000 7f39aef2d640 1 -- start start 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39aef2d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998003b70 0x7f39a80cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39aef2d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 0x7f39a80ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39aef2d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a80caed0 0x7f39a80b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39aef2d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39a80b0940 con 0x7f39a814a0a0 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39aef2d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f39a80cd120 con 0x7f39a80caed0 2024-04-25T15:22:27.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39aef2d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f39a80b0ac0 con 0x7f3998003b70 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39a7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 0x7f39a80ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39a7fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 0x7f39a80ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33722/0 (socket says 172.21.15.146:33722) 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.578+0000 7f39a7fff640 1 -- 172.21.15.146:0/3528622509 learned_addr learned my addr 172.21.15.146:0/3528622509 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39ad4ca640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a80caed0 0x7f39a80b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39accc9640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998003b70 0x7f39a80cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a7fff640 1 -- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998003b70 msgr2=0x7f39a80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a7fff640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998003b70 0x7f39a80cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a7fff640 1 -- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a80caed0 msgr2=0x7f39a80b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a7fff640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a80caed0 0x7f39a80b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a7fff640 1 -- 172.21.15.146:0/3528622509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a80cb700 con 0x7f39a814a0a0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a7fff640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 0x7f39a80ca990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3994004f40 tx=0x7f39940074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400f750 con 0x7f39a814a0a0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3994018070 con 0x7f39a814a0a0 2024-04-25T15:22:27.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a80cb9e0 con 0x7f39a814a0a0 2024-04-25T15:22:27.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400fb70 con 0x7f39a814a0a0 2024-04-25T15:22:27.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.579+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a80b46e0 con 0x7f39a814a0a0 2024-04-25T15:22:27.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.580+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a80ae7d0 con 0x7f39a814a0a0 2024-04-25T15:22:27.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.581+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f399400fd10 con 0x7f39a814a0a0 2024-04-25T15:22:27.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.581+0000 7f39a57fa640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f398c074540 0x7f398c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:27.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.581+0000 7f39accc9640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f398c074540 0x7f398c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:27.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.581+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6159+0+0 (secure 0 0 0) 0x7f39940953b0 con 0x7f39a814a0a0 2024-04-25T15:22:27.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.581+0000 7f39accc9640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f398c074540 0x7f398c076a50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f399c032000 tx=0x7f399c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:27.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.586+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39a80ae7d0 con 0x7f39a814a0a0 2024-04-25T15:22:27.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.800+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f39a80cb490 con 0x7f39a814a0a0 2024-04-25T15:22:27.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.801+0000 7f39a57fa640 1 -- 172.21.15.146:0/3528622509 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7f399405f990 con 0x7f39a814a0a0 2024-04-25T15:22:27.802 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.802+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f398c074540 msgr2=0x7f398c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.802+0000 7f39aef2d640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f398c074540 0x7f398c076a50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f399c032000 tx=0x7f399c009030 comp rx=0 tx=0).stop 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.802+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 msgr2=0x7f39a80ca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.802+0000 7f39aef2d640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 0x7f39a80ca990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3994004f40 tx=0x7f39940074f0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 shutdown_connections 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f398c074540 0x7f398c076a50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39a80caed0 0x7f39a80b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39a814a0a0 0x7f39a80ca990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 --2- 172.21.15.146:0/3528622509 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998003b70 0x7f39a80cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 >> 172.21.15.146:0/3528622509 conn(0x7f39a80ce0c0 msgr2=0x7f39a815beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 shutdown_connections 2024-04-25T15:22:27.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:27.803+0000 7f39aef2d640 1 -- 172.21.15.146:0/3528622509 wait complete. 2024-04-25T15:22:27.815 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:22:27.815 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:22:27.816 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:22:27.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.816 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:22:27.835 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:27.836 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42586sec preferred_lft 42586sec 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:27.837 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link 2024-04-25T15:22:27.838 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.838 INFO:teuthology.orchestra.run.smithi177.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:27.838 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:27.838 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::7cc7:d2ff:fef5:71b6/64 scope link 2024-04-25T15:22:27.838 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:27.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.838 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:22:27.903 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:22:27.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:27.904 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:22:27.963 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:22:27.963 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:22:27.963 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:22:27.964 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:22:27.964 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:22:27.964 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:22:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:22:28.088 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:28.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.095 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:22:28.110 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:22:28.110 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:22:28.111 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:22:28.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-04-25T15:22:28.175 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-25T15:22:28.175 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:28.175 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:28.230 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:22:28.230 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:28.230 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:28.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:22:28.350 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:28.405 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:22:28.405 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:22:28.406 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:22:28.407 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:22:28.408 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:28.408 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:28.408 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:28.408 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:28.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.408 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:22:28.460 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:22:28.461 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-25T15:22:28.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:22:28.525 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:22:28.585 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:28.586 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:28.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.588 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:22:28.592 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:2024-04-25T15:22:28.591+0000 7f1bee329480 -1 init, newargv = 0x7f1bdc054030 newargc=15 2024-04-25T15:22:28.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[43864]: starting ceph client 2024-04-25T15:22:28.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[43864]: starting fuse 2024-04-25T15:22:28.609 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:22:28.609 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:22:28.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:22:28.609 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:22:28.610 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.1")) 2024-04-25T15:22:28.611 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:22:28.727 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.1.43864.asok 2024-04-25T15:22:28.730 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:28.737 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43864.asok 2024-04-25T15:22:28.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43864.asok status 2024-04-25T15:22:28.878 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "1", 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "43864", 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/" 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:28.879 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 5555, 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 5555 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 1807574668 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:28.880 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 1807574668 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.5555 192.168.0.1:0/1807574668", 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.1:0/1807574668", 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 1, 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 30, 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 56, 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:22:28.881 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:22:28.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.887 DEBUG:teuthology.orchestra.run.smithi177:> stat --file-system '--printf=%T 2024-04-25T15:22:28.887 DEBUG:teuthology.orchestra.run.smithi177:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:28.902 INFO:teuthology.orchestra.run.smithi177.stdout:fuseblk 2024-04-25T15:22:28.902 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:28.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:28.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:28.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 25 2024-04-25T15:22:29.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.085+0000 7facf8364640 1 Processor -- start 2024-04-25T15:22:29.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.085+0000 7facf8364640 1 -- start start 2024-04-25T15:22:29.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf8364640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00b4ca0 0x7facf00b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf8364640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facf00ce6f0 0x7facf00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf8364640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf008d590 con 0x7facf008dd50 2024-04-25T15:22:29.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf8364640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7facf008d8c0 con 0x7facf00ce6f0 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf8364640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7facf00d27d0 con 0x7facf00b4ca0 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf6901640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facf00ce6f0 0x7facf00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf6901640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facf00ce6f0 0x7facf00ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33730/0 (socket says 172.21.15.146:33730) 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf6901640 1 -- 172.21.15.146:0/3693776616 learned_addr learned my addr 172.21.15.146:0/3693776616 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.086+0000 7facf58ff640 1 --2- 172.21.15.146:0/3693776616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00b4ca0 0x7facf00b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf6901640 1 -- 172.21.15.146:0/3693776616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00b4ca0 msgr2=0x7facf00b8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf6901640 1 --2- 172.21.15.146:0/3693776616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00b4ca0 0x7facf00b8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf6901640 1 -- 172.21.15.146:0/3693776616 >> v1:172.21.15.177:6789/0 conn(0x7facf008dd50 legacy=0x7facf00d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf6901640 1 -- 172.21.15.146:0/3693776616 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf00b48b0 con 0x7facf00ce6f0 2024-04-25T15:22:29.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf6901640 1 --2- 172.21.15.146:0/3693776616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00ce6f0 0x7facf00ceb00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7face8009870 tx=0x7face802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5b2284e105cbf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf50fe640 1 -- 172.21.15.146:0/3693776616 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8037070 con 0x7facf00ce6f0 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.087+0000 7facf50fe640 1 -- 172.21.15.146:0/3693776616 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7face8032440 con 0x7facf00ce6f0 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf50fe640 1 -- 172.21.15.146:0/3693776616 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8032810 con 0x7facf00ce6f0 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 -- 172.21.15.146:0/3693776616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00ce6f0 msgr2=0x7facf00ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 --2- 172.21.15.146:0/3693776616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00ce6f0 0x7facf00ceb00 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7face8009870 tx=0x7face802a270 comp rx=0 tx=0).stop 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 -- 172.21.15.146:0/3693776616 shutdown_connections 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 --2- 172.21.15.146:0/3693776616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00ce6f0 0x7facf00ceb00 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 --2- 172.21.15.146:0/3693776616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00b4ca0 0x7facf00b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 -- 172.21.15.146:0/3693776616 >> 172.21.15.146:0/3693776616 conn(0x7facf00b5430 msgr2=0x7facf00b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 -- 172.21.15.146:0/3693776616 shutdown_connections 2024-04-25T15:22:29.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.088+0000 7facf8364640 1 -- 172.21.15.146:0/3693776616 wait complete. 2024-04-25T15:22:29.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.090+0000 7facf8364640 1 Processor -- start 2024-04-25T15:22:29.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.090+0000 7facf8364640 1 -- start start 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf8364640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 0x7facf01dfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf8364640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00ce6f0 0x7facf01f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf8364640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facf01e0a90 0x7facf00b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf8364640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7facf00d24c0 con 0x7facf00b4ca0 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf8364640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7facf00d2340 con 0x7facf01e0a90 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf8364640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7facf00cf040 con 0x7facf00ce6f0 2024-04-25T15:22:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.091+0000 7facf6100640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 0x7facf01dfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 0x7facf01dfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33742/0 (socket says 172.21.15.146:33742) 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 -- 172.21.15.146:0/1631920658 learned_addr learned my addr 172.21.15.146:0/1631920658 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf58ff640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00ce6f0 0x7facf01f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6901640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facf01e0a90 0x7facf00b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 -- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00ce6f0 msgr2=0x7facf01f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00ce6f0 0x7facf01f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 -- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facf01e0a90 msgr2=0x7facf00b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facf01e0a90 0x7facf00b8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 -- 172.21.15.146:0/1631920658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf00b7d50 con 0x7facf00b4ca0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf6100640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 0x7facf01dfbb0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7facec00c690 tx=0x7facec00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec003b70 con 0x7facf00b4ca0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec003d30 con 0x7facf00b4ca0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf00b8010 con 0x7facf00b4ca0 2024-04-25T15:22:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec005310 con 0x7facf00b4ca0 2024-04-25T15:22:29.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.092+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf00b85d0 con 0x7facf00b4ca0 2024-04-25T15:22:29.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.093+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf006a0e0 con 0x7facf00b4ca0 2024-04-25T15:22:29.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.094+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7facec0054b0 con 0x7facf00b4ca0 2024-04-25T15:22:29.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.094+0000 7face6ffd640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facd4074540 0x7facd4076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.095+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7facec094ff0 con 0x7facf00b4ca0 2024-04-25T15:22:29.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.095+0000 7facf58ff640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facd4074540 0x7facd4076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.095+0000 7facf58ff640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facd4074540 0x7facd4076a50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7facf01f7560 tx=0x7face0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.099+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7facec05f0a0 con 0x7facf00b4ca0 2024-04-25T15:22:29.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.277+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7facf00d0a90 con 0x7facf00b4ca0 2024-04-25T15:22:29.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.280+0000 7face6ffd640 1 -- 172.21.15.146:0/1631920658 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v17)=0 v17) ==== 108+0+0 (secure 0 0 0) 0x7facec0633a0 con 0x7facf00b4ca0 2024-04-25T15:22:29.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facd4074540 msgr2=0x7facd4076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facd4074540 0x7facd4076a50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7facf01f7560 tx=0x7face0009190 comp rx=0 tx=0).stop 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 msgr2=0x7facf01dfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 0x7facf01dfbb0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7facec00c690 tx=0x7facec00cb50 comp rx=0 tx=0).stop 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 shutdown_connections 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7facd4074540 0x7facd4076a50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facf01e0a90 0x7facf00b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facf00ce6f0 0x7facf01f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 --2- 172.21.15.146:0/1631920658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf00b4ca0 0x7facf01dfbb0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 >> 172.21.15.146:0/1631920658 conn(0x7facf00b5430 msgr2=0x7facf0072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 shutdown_connections 2024-04-25T15:22:29.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.283+0000 7facf8364640 1 -- 172.21.15.146:0/1631920658 wait complete. 2024-04-25T15:22:29.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_cache_quiesce_splitauth true 2024-04-25T15:22:29.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.404+0000 7fbdc2df3640 1 Processor -- start 2024-04-25T15:22:29.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.404+0000 7fbdc2df3640 1 -- start start 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdc2df3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdbc071d30 0x7fbdbc0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdc2df3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbdbc0d3300 0x7fbdbc0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdc2df3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdbc0d2260 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdc2df3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbdbc0d2540 con 0x7fbdbc071d30 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdc2df3640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbdbc0d2820 con 0x7fbdbc0d3300 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdbc071d30 0x7fbdbc0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdbc071d30 0x7fbdbc0d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33754/0 (socket says 172.21.15.146:33754) 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 -- 172.21.15.146:0/197918574 learned_addr learned my addr 172.21.15.146:0/197918574 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdc1390640 1 --2- 172.21.15.146:0/197918574 >> v2:172.21.15.177:3300/0 conn(0x7fbdbc0d3300 0x7fbdbc0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 -- 172.21.15.146:0/197918574 >> v2:172.21.15.177:3300/0 conn(0x7fbdbc0d3300 msgr2=0x7fbdbc0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 --2- 172.21.15.146:0/197918574 >> v2:172.21.15.177:3300/0 conn(0x7fbdbc0d3300 0x7fbdbc0cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 -- 172.21.15.146:0/197918574 >> v1:172.21.15.177:6789/0 conn(0x7fbdbc0b2cd0 legacy=0x7fbdbc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:29.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.406+0000 7fbdbbfff640 1 -- 172.21.15.146:0/197918574 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc0d2b00 con 0x7fbdbc071d30 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.407+0000 7fbdbbfff640 1 --2- 172.21.15.146:0/197918574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc071d30 0x7fbdbc0d1c80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc08e060 tx=0x7fbdb002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb995b2653fc6df6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.407+0000 7fbdbb7fe640 1 -- 172.21.15.146:0/197918574 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0037070 con 0x7fbdbc071d30 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.407+0000 7fbdbb7fe640 1 -- 172.21.15.146:0/197918574 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb0032440 con 0x7fbdbc071d30 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.407+0000 7fbdbb7fe640 1 -- 172.21.15.146:0/197918574 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0032810 con 0x7fbdbc071d30 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 -- 172.21.15.146:0/197918574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc071d30 msgr2=0x7fbdbc0d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/197918574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc071d30 0x7fbdbc0d1c80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc08e060 tx=0x7fbdb002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 -- 172.21.15.146:0/197918574 shutdown_connections 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/197918574 >> v2:172.21.15.177:3300/0 conn(0x7fbdbc0d3300 0x7fbdbc0cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/197918574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc071d30 0x7fbdbc0d1c80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 -- 172.21.15.146:0/197918574 >> 172.21.15.146:0/197918574 conn(0x7fbdbc0b52e0 msgr2=0x7fbdbc0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 -- 172.21.15.146:0/197918574 shutdown_connections 2024-04-25T15:22:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.408+0000 7fbdc2df3640 1 -- 172.21.15.146:0/197918574 wait complete. 2024-04-25T15:22:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.409+0000 7fbdc2df3640 1 Processor -- start 2024-04-25T15:22:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.409+0000 7fbdc2df3640 1 -- start start 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc2df3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 0x7fbdbc14e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc2df3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdbc0d3300 0x7fbdbc1f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc2df3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdbc1f6360 0x7fbdbc162c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc2df3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc08dbf0 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc2df3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc0cfb70 con 0x7fbdbc0d3300 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc2df3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbdbc08dd70 con 0x7fbdbc1f6360 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 0x7fbdbc14e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 0x7fbdbc14e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33770/0 (socket says 172.21.15.146:33770) 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 -- 172.21.15.146:0/3745143553 learned_addr learned my addr 172.21.15.146:0/3745143553 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:29.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc1390640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdbc1f6360 0x7fbdbc162c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdbbfff640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdbc0d3300 0x7fbdbc1f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 -- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdbc1f6360 msgr2=0x7fbdbc162c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdbc1f6360 0x7fbdbc162c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 -- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdbc0d3300 msgr2=0x7fbdbc1f5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdbc0d3300 0x7fbdbc1f5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.411+0000 7fbdc0b8f640 1 -- 172.21.15.146:0/3745143553 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc0cbc80 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.412+0000 7fbdc0b8f640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 0x7fbdbc14e350 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009fd0 tx=0x7fbdac00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.412+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac015030 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.412+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdac004020 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.412+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdbc1f6b50 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.412+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac004450 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.412+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdbc0b67b0 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.413+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdbc0b62c0 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.413+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7fbdac018070 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.414+0000 7fbdb97fa640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbda8074540 0x7fbda8076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.414+0000 7fbdbbfff640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbda8074540 0x7fbda8076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.414+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fbdac094aa0 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.414+0000 7fbdbbfff640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbda8074540 0x7fbda8076a50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0034000 tx=0x7fbdb002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.418+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbdac05e9b0 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.594+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mds_cache_quiesce_splitauth}] v 0) -- 0x7fbdbc06a0e0 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.597+0000 7fbdb97fa640 1 -- 172.21.15.146:0/3745143553 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_cache_quiesce_splitauth}]=0 v18)=0 v18) ==== 128+0+0 (secure 0 0 0) 0x7fbdac067080 con 0x7fbdbc0b2cd0 2024-04-25T15:22:29.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.599+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbda8074540 msgr2=0x7fbda8076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.599+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbda8074540 0x7fbda8076a50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0034000 tx=0x7fbdb002e000 comp rx=0 tx=0).stop 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.599+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 msgr2=0x7fbdbc14e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.599+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 0x7fbdbc14e350 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009fd0 tx=0x7fbdac00b870 comp rx=0 tx=0).stop 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 shutdown_connections 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbda8074540 0x7fbda8076a50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdbc1f6360 0x7fbdbc162c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdbc0d3300 0x7fbdbc1f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 --2- 172.21.15.146:0/3745143553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdbc0b2cd0 0x7fbdbc14e350 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 >> 172.21.15.146:0/3745143553 conn(0x7fbdbc0b52e0 msgr2=0x7fbdbc0723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 shutdown_connections 2024-04-25T15:22:29.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.600+0000 7fbdc2df3640 1 -- 172.21.15.146:0/3745143553 wait complete. 2024-04-25T15:22:29.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol_quiesce --mode=777 2024-04-25T15:22:29.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.721+0000 7f24d64e0640 1 Processor -- start 2024-04-25T15:22:29.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.722+0000 7f24d64e0640 1 -- start start 2024-04-25T15:22:29.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24d64e0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f24d00b3060 0x7f24d00b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24d64e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24d00d1e00 0x7f24d00d2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24d64e0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d00d2750 con 0x7f24d015de20 2024-04-25T15:22:29.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24d64e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f24d006c880 con 0x7f24d00d1e00 2024-04-25T15:22:29.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24d64e0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f24d006cb60 con 0x7f24d00b3060 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24d00d1e00 0x7f24d00d2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24d00d1e00 0x7f24d00d2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33784/0 (socket says 172.21.15.146:33784) 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 -- 172.21.15.146:0/3073183757 learned_addr learned my addr 172.21.15.146:0/3073183757 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cffff640 1 --2- 172.21.15.146:0/3073183757 >> v2:172.21.15.177:3300/0 conn(0x7f24d00b3060 0x7f24d00b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 -- 172.21.15.146:0/3073183757 >> v2:172.21.15.177:3300/0 conn(0x7f24d00b3060 msgr2=0x7f24d00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 --2- 172.21.15.146:0/3073183757 >> v2:172.21.15.177:3300/0 conn(0x7f24d00b3060 0x7f24d00b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 -- 172.21.15.146:0/3073183757 >> v1:172.21.15.177:6789/0 conn(0x7f24d015de20 legacy=0x7f24d015e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cf7fe640 1 -- 172.21.15.146:0/3073183757 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d006ce40 con 0x7f24d00d1e00 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.723+0000 7f24cffff640 1 --2- 172.21.15.146:0/3073183757 >> v2:172.21.15.177:3300/0 conn(0x7f24d00b3060 0x7f24d00b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:29.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.724+0000 7f24cf7fe640 1 --2- 172.21.15.146:0/3073183757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d00d2210 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f24d00b3a60 tx=0x7f24c402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374cba73bf97fd8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.724+0000 7f24ceffd640 1 -- 172.21.15.146:0/3073183757 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4037070 con 0x7f24d00d1e00 2024-04-25T15:22:29.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.724+0000 7f24ceffd640 1 -- 172.21.15.146:0/3073183757 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c4032440 con 0x7f24d00d1e00 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.724+0000 7f24ceffd640 1 -- 172.21.15.146:0/3073183757 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4032810 con 0x7f24d00d1e00 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 -- 172.21.15.146:0/3073183757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 msgr2=0x7f24d00d2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 --2- 172.21.15.146:0/3073183757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d00d2210 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f24d00b3a60 tx=0x7f24c402a2c0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 -- 172.21.15.146:0/3073183757 shutdown_connections 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 --2- 172.21.15.146:0/3073183757 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d00d2210 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 --2- 172.21.15.146:0/3073183757 >> v2:172.21.15.177:3300/0 conn(0x7f24d00b3060 0x7f24d00b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 -- 172.21.15.146:0/3073183757 >> 172.21.15.146:0/3073183757 conn(0x7f24d009b2b0 msgr2=0x7f24d00b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 -- 172.21.15.146:0/3073183757 shutdown_connections 2024-04-25T15:22:29.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.725+0000 7f24d64e0640 1 -- 172.21.15.146:0/3073183757 wait complete. 2024-04-25T15:22:29.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.727+0000 7f24d64e0640 1 Processor -- start 2024-04-25T15:22:29.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.727+0000 7f24d64e0640 1 -- start start 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d64e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24d00b3060 0x7f24d01f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d64e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d01f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d64e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24d015de20 0x7f24d015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d64e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f24d006c550 con 0x7f24d00d1e00 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d64e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24d006d400 con 0x7f24d00b3060 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d64e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f24d006d920 con 0x7f24d015de20 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24cf7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d01f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24cf7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d01f5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33790/0 (socket says 172.21.15.146:33790) 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24cf7fe640 1 -- 172.21.15.146:0/453777658 learned_addr learned my addr 172.21.15.146:0/453777658 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24cffff640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24d00b3060 0x7f24d01f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.728+0000 7f24d4a7d640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24d015de20 0x7f24d015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24cf7fe640 1 -- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24d015de20 msgr2=0x7f24d015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24cf7fe640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24d015de20 0x7f24d015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24cf7fe640 1 -- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24d00b3060 msgr2=0x7f24d01f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24cf7fe640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24d00b3060 0x7f24d01f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24cf7fe640 1 -- 172.21.15.146:0/453777658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d00ca4e0 con 0x7f24d00d1e00 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24d4a7d640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24d015de20 0x7f24d015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24cf7fe640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d01f5ac0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f24c4000c00 tx=0x7f24c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4041020 con 0x7f24d00d1e00 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c4030970 con 0x7f24d00d1e00 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4037040 con 0x7f24d00d1e00 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d00ca7c0 con 0x7f24d00d1e00 2024-04-25T15:22:29.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.729+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d00d1180 con 0x7f24d00d1e00 2024-04-25T15:22:29.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.730+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d00ca250 con 0x7f24d00d1e00 2024-04-25T15:22:29.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.731+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f24c4044020 con 0x7f24d00d1e00 2024-04-25T15:22:29.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.731+0000 7f24ccff9640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24bc074540 0x7f24bc076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:29.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.731+0000 7f24cffff640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24bc074540 0x7f24bc076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:29.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.731+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f24c40b5db0 con 0x7f24d00d1e00 2024-04-25T15:22:29.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.732+0000 7f24cffff640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24bc074540 0x7f24bc076a50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f24c000a950 tx=0x7f24c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:29.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.736+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24c404b030 con 0x7f24d00d1e00 2024-04-25T15:22:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.937+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f24d006a0e0 con 0x7f24bc074540 2024-04-25T15:22:29.943 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:22:29.942+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:22:29.943 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:22:29.942+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:22:29.943 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:22:29.942+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:22:29.943 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:22:29.942+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:22:29.943 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-04-25T15:22:29.942+0000 7f8f661cf640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:22:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.969+0000 7f24ccff9640 1 -- 172.21.15.146:0/453777658 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f24d006a0e0 con 0x7f24bc074540 2024-04-25T15:22:29.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24bc074540 msgr2=0x7f24bc076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24bc074540 0x7f24bc076a50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f24c000a950 tx=0x7f24c0008040 comp rx=0 tx=0).stop 2024-04-25T15:22:29.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 msgr2=0x7f24d01f5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:29.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d01f5ac0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f24c4000c00 tx=0x7f24c40046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 shutdown_connections 2024-04-25T15:22:29.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f24bc074540 0x7f24bc076a50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24d015de20 0x7f24d015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d00d1e00 0x7f24d01f5ac0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 --2- 172.21.15.146:0/453777658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24d00b3060 0x7f24d01f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:29.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 >> 172.21.15.146:0/453777658 conn(0x7f24d009b2b0 msgr2=0x7f24d00b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:29.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 shutdown_connections 2024-04-25T15:22:29.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:29.971+0000 7f24d64e0640 1 -- 172.21.15.146:0/453777658 wait complete. 2024-04-25T15:22:29.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_quiesce 2024-04-25T15:22:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.093+0000 7f0c92a77640 1 Processor -- start 2024-04-25T15:22:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.093+0000 7f0c92a77640 1 -- start start 2024-04-25T15:22:30.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.094+0000 7f0c92a77640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c08dd50 0x7f0c8c0b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.094+0000 7f0c92a77640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c08d590 con 0x7f0c8c0b8f30 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.094+0000 7f0c92a77640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c0d27d0 con 0x7f0c8c0ce6f0 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.094+0000 7f0c92a77640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0c8c0b9340 con 0x7f0c8c08dd50 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.094+0000 7f0c8bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0c8c0b8f30 0x7f0c8c0d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50258/0 (socket says 172.21.15.146:50258) 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.094+0000 7f0c8bfff640 1 -- 172.21.15.146:0/3562880153 learned_addr learned my addr 172.21.15.146:0/3562880153 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c90813640 1 --2- 172.21.15.146:0/3562880153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c08dd50 0x7f0c8c0b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303115982 0 0) 0x7f0c8c08d590 con 0x7f0c8c0b8f30 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c7c003520 con 0x7f0c8c0b8f30 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529418118 0 0) 0x7f0c8c0d27d0 con 0x7f0c8c0ce6f0 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c8c08d590 con 0x7f0c8c0ce6f0 2024-04-25T15:22:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c90813640 1 -- 172.21.15.146:0/3562880153 >> v1:172.21.15.177:6789/0 conn(0x7f0c8c0ce6f0 legacy=0x7f0c8c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2479622406 0 0) 0x7f0c7c003520 con 0x7f0c8c0b8f30 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c90813640 1 -- 172.21.15.146:0/3562880153 >> v1:172.21.15.146:6789/0 conn(0x7f0c8c0b8f30 legacy=0x7f0c8c0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.095+0000 7f0c90813640 1 -- 172.21.15.146:0/3562880153 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c0b48b0 con 0x7f0c8c08dd50 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.096+0000 7f0c90813640 1 --2- 172.21.15.146:0/3562880153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c08dd50 0x7f0c8c0b4ca0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0c800097d0 tx=0x7f0c8002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10b5033df4fbfcb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.096+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80037070 con 0x7f0c8c08dd50 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.096+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8002ac30 con 0x7f0c8c08dd50 2024-04-25T15:22:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.096+0000 7f0c8b7fe640 1 -- 172.21.15.146:0/3562880153 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78001670 con 0x7f0c8c08dd50 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 -- 172.21.15.146:0/3562880153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c08dd50 msgr2=0x7f0c8c0b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 --2- 172.21.15.146:0/3562880153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c08dd50 0x7f0c8c0b4ca0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0c800097d0 tx=0x7f0c8002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 -- 172.21.15.146:0/3562880153 shutdown_connections 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 --2- 172.21.15.146:0/3562880153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c08dd50 0x7f0c8c0b4ca0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 -- 172.21.15.146:0/3562880153 >> 172.21.15.146:0/3562880153 conn(0x7f0c8c0b5430 msgr2=0x7f0c8c0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 -- 172.21.15.146:0/3562880153 shutdown_connections 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.097+0000 7f0c92a77640 1 -- 172.21.15.146:0/3562880153 wait complete. 2024-04-25T15:22:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.098+0000 7f0c92a77640 1 Processor -- start 2024-04-25T15:22:30.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.099+0000 7f0c92a77640 1 -- start start 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c92a77640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c8c0b8f30 0x7f0c8c1e7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c92a77640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c0ce6f0 0x7f0c8c1e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c92a77640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 0x7f0c8c1f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c92a77640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c0d2340 con 0x7f0c8c1f7040 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c92a77640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c08d8c0 con 0x7f0c8c0b8f30 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c92a77640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f0c8c0d24c0 con 0x7f0c8c0ce6f0 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 0x7f0c8c1f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 0x7f0c8c1f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33834/0 (socket says 172.21.15.146:33834) 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 -- 172.21.15.146:0/1847199140 learned_addr learned my addr 172.21.15.146:0/1847199140 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c90813640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c8c0b8f30 0x7f0c8c1e7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c8bfff640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c0ce6f0 0x7f0c8c1e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 -- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c0ce6f0 msgr2=0x7f0c8c1e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c0ce6f0 0x7f0c8c1e5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 -- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c8c0b8f30 msgr2=0x7f0c8c1e7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c8c0b8f30 0x7f0c8c1e7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.100+0000 7f0c91014640 1 -- 172.21.15.146:0/1847199140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c1f7990 con 0x7f0c8c1f7040 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.101+0000 7f0c91014640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 0x7f0c8c1f7450 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c6e0 tx=0x7f0c8400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.101+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84003b70 con 0x7f0c8c1f7040 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.101+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84003d10 con 0x7f0c8c1f7040 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.101+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8c1e86b0 con 0x7f0c8c1f7040 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.101+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840052d0 con 0x7f0c8c1f7040 2024-04-25T15:22:30.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.101+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c8c1e8c70 con 0x7f0c8c1f7040 2024-04-25T15:22:30.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.102+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c8c06a0e0 con 0x7f0c8c1f7040 2024-04-25T15:22:30.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.102+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f0c84005470 con 0x7f0c8c1f7040 2024-04-25T15:22:30.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.103+0000 7f0c897fa640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0c70074540 0x7f0c70076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.103+0000 7f0c90813640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0c70074540 0x7f0c70076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.103+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f0c84095300 con 0x7f0c8c1f7040 2024-04-25T15:22:30.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.103+0000 7f0c90813640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0c70074540 0x7f0c70076a50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0c80034000 tx=0x7f0c8002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.107+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c8c1f7990 con 0x7f0c8c1f7040 2024-04-25T15:22:30.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.313+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "target": ["mon-mgr", ""]}) -- 0x7f0c8c0b8b30 con 0x7f0c70074540 2024-04-25T15:22:30.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.317+0000 7f0c897fa640 1 -- 172.21.15.146:0/1847199140 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+70 (secure 0 0 0) 0x7f0c8c0b8b30 con 0x7f0c70074540 2024-04-25T15:22:30.319 INFO:teuthology.orchestra.run.smithi146.stdout:/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0c70074540 msgr2=0x7f0c70076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0c70074540 0x7f0c70076a50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0c80034000 tx=0x7f0c8002e000 comp rx=0 tx=0).stop 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 msgr2=0x7f0c8c1f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 0x7f0c8c1f7450 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c6e0 tx=0x7f0c8400cba0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 shutdown_connections 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0c70074540 0x7f0c70076a50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c8c1f7040 0x7f0c8c1f7450 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c8c0ce6f0 0x7f0c8c1e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 --2- 172.21.15.146:0/1847199140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c8c0b8f30 0x7f0c8c1e7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 >> 172.21.15.146:0/1847199140 conn(0x7f0c8c0b5430 msgr2=0x7f0c8c0cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:30.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 shutdown_connections 2024-04-25T15:22:30.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.319+0000 7f0c92a77640 1 -- 172.21.15.146:0/1847199140 wait complete. 2024-04-25T15:22:30.332 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_size 10 2024-04-25T15:22:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.442+0000 7f9a15642640 1 Processor -- start 2024-04-25T15:22:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.442+0000 7f9a15642640 1 -- start start 2024-04-25T15:22:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a15642640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a100b2cd0 0x7f9a100d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a15642640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a10071d30 0x7f9a100cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a15642640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a100d2260 con 0x7f9a100d3300 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a15642640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9a100d2540 con 0x7f9a100b2cd0 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a15642640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9a100d2820 con 0x7f9a10071d30 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a0effd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a100b2cd0 0x7f9a100d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a0effd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a100b2cd0 0x7f9a100d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33848/0 (socket says 172.21.15.146:33848) 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.443+0000 7f9a0effd640 1 -- 172.21.15.146:0/1142894758 learned_addr learned my addr 172.21.15.146:0/1142894758 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.444+0000 7f9a0f7fe640 1 --2- 172.21.15.146:0/1142894758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a10071d30 0x7f9a100cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.444+0000 7f9a0effd640 1 -- 172.21.15.146:0/1142894758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a10071d30 msgr2=0x7f9a100cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.444+0000 7f9a0effd640 1 --2- 172.21.15.146:0/1142894758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a10071d30 0x7f9a100cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.444+0000 7f9a0effd640 1 -- 172.21.15.146:0/1142894758 >> v1:172.21.15.177:6789/0 conn(0x7f9a100d3300 legacy=0x7f9a10071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.444+0000 7f9a0effd640 1 -- 172.21.15.146:0/1142894758 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a100d2b00 con 0x7f9a100b2cd0 2024-04-25T15:22:30.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.444+0000 7f9a0effd640 1 --2- 172.21.15.146:0/1142894758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a100d1c80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9a00009850 tx=0x7f9a0002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c966a2b1c04db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a0e7fc640 1 -- 172.21.15.146:0/1142894758 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00004030 con 0x7f9a100b2cd0 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a0e7fc640 1 -- 172.21.15.146:0/1142894758 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a000041d0 con 0x7f9a100b2cd0 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a0e7fc640 1 -- 172.21.15.146:0/1142894758 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00003610 con 0x7f9a100b2cd0 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a15642640 1 -- 172.21.15.146:0/1142894758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 msgr2=0x7f9a100d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a15642640 1 --2- 172.21.15.146:0/1142894758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a100d1c80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9a00009850 tx=0x7f9a0002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a15642640 1 -- 172.21.15.146:0/1142894758 shutdown_connections 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a15642640 1 --2- 172.21.15.146:0/1142894758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a10071d30 0x7f9a100cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a15642640 1 --2- 172.21.15.146:0/1142894758 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a100d1c80 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.445+0000 7f9a15642640 1 -- 172.21.15.146:0/1142894758 >> 172.21.15.146:0/1142894758 conn(0x7f9a100b52e0 msgr2=0x7f9a100b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.446+0000 7f9a15642640 1 -- 172.21.15.146:0/1142894758 shutdown_connections 2024-04-25T15:22:30.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.446+0000 7f9a15642640 1 -- 172.21.15.146:0/1142894758 wait complete. 2024-04-25T15:22:30.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.447+0000 7f9a15642640 1 Processor -- start 2024-04-25T15:22:30.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.447+0000 7f9a15642640 1 -- start start 2024-04-25T15:22:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a10151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a100d3300 0x7f9a100cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a100c9e30 0x7f9a10153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1008dbf0 con 0x7f9a100b2cd0 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a100cfb70 con 0x7f9a100d3300 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9a1008dd70 con 0x7f9a100c9e30 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a10151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a10151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33860/0 (socket says 172.21.15.146:33860) 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 -- 172.21.15.146:0/3359453152 learned_addr learned my addr 172.21.15.146:0/3359453152 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a07fff640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a100d3300 0x7f9a100cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0f7fe640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a100c9e30 0x7f9a10153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 -- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a100c9e30 msgr2=0x7f9a10153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a100c9e30 0x7f9a10153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 -- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a100d3300 msgr2=0x7f9a100cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a100d3300 0x7f9a100cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 -- 172.21.15.146:0/3359453152 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a100c8d90 con 0x7f9a100b2cd0 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a0effd640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a10151b20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9a0002a1e0 tx=0x7f9a000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00041020 con 0x7f9a100b2cd0 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a00003c50 con 0x7f9a100b2cd0 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a100c9070 con 0x7f9a100b2cd0 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00004030 con 0x7f9a100b2cd0 2024-04-25T15:22:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.449+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a100c9630 con 0x7f9a100b2cd0 2024-04-25T15:22:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.451+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1006a0e0 con 0x7f9a100b2cd0 2024-04-25T15:22:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.451+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 6) ==== 96833+0+0 (secure 0 0 0) 0x7f9a00044020 con 0x7f9a100b2cd0 2024-04-25T15:22:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.452+0000 7f9a077fe640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99f0074540 0x7f99f0076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.452+0000 7f9a07fff640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99f0074540 0x7f99f0076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.452+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f9a000b5f80 con 0x7f9a100b2cd0 2024-04-25T15:22:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.452+0000 7f9a07fff640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99f0074540 0x7f99f0076a50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9a100b2770 tx=0x7f99f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.456+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a0004b030 con 0x7f9a100b2cd0 2024-04-25T15:22:30.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.608+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f9a0007c0f0 con 0x7f9a100b2cd0 2024-04-25T15:22:30.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.635+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f9a1015f3b0 con 0x7f9a100b2cd0 2024-04-25T15:22:30.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.638+0000 7f9a077fe640 1 -- 172.21.15.146:0/3359453152 <== mon.0 v2:172.21.15.146:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v19)=0 v19) ==== 117+0+0 (secure 0 0 0) 0x7f9a00080560 con 0x7f9a100b2cd0 2024-04-25T15:22:30.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99f0074540 msgr2=0x7f99f0076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99f0074540 0x7f99f0076a50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9a100b2770 tx=0x7f99f8009040 comp rx=0 tx=0).stop 2024-04-25T15:22:30.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 msgr2=0x7f9a10151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a10151b20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9a0002a1e0 tx=0x7f9a000046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 shutdown_connections 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f99f0074540 0x7f99f0076a50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a100c9e30 0x7f9a10153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a100d3300 0x7f9a100cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 --2- 172.21.15.146:0/3359453152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a100b2cd0 0x7f9a10151b20 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 >> 172.21.15.146:0/3359453152 conn(0x7f9a100b52e0 msgr2=0x7f9a100b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 shutdown_connections 2024-04-25T15:22:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.640+0000 7f9a15642640 1 -- 172.21.15.146:0/3359453152 wait complete. 2024-04-25T15:22:30.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-04-25T15:22:30.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.763+0000 7f44286e7640 1 Processor -- start 2024-04-25T15:22:30.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.763+0000 7f44286e7640 1 -- start start 2024-04-25T15:22:30.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f44286e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 0x7f44200b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f44286e7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f44200d10b0 0x7f442015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f44286e7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44200b0520 con 0x7f442006d560 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f44286e7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f44200b0850 con 0x7f44200d10b0 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f44286e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4420070d90 con 0x7f4420070500 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4426c84640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f442006d560 0x7f442006d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32792/0 (socket says 172.21.15.146:32792) 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4426c84640 1 -- 172.21.15.146:0/1788013566 learned_addr learned my addr 172.21.15.146:0/1788013566 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2192553693 0 0) 0x7f44200b0520 con 0x7f442006d560 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4426483640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 0x7f44200b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f440c003520 con 0x7f442006d560 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425c82640 1 --2- 172.21.15.146:0/1788013566 >> v2:172.21.15.177:3300/0 conn(0x7f44200d10b0 0x7f442015cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 44545968 0 0) 0x7f440c003520 con 0x7f442006d560 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 msgr2=0x7f44200b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 0x7f44200b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 >> v2:172.21.15.177:3300/0 conn(0x7f44200d10b0 msgr2=0x7f442015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 --2- 172.21.15.146:0/1788013566 >> v2:172.21.15.177:3300/0 conn(0x7f44200d10b0 0x7f442015cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4420071070 con 0x7f442006d560 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.765+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f4418001a10 con 0x7f442006d560 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 >> v1:172.21.15.146:6789/0 conn(0x7f442006d560 legacy=0x7f442006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 0x7f440c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f440c003f30 0x7f440c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f440c004d40 0x7f440c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f44200b0850 con 0x7f440c003f30 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4420070d90 con 0x7f440c004d40 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f440c00ba10 con 0x7f4420070500 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425c82640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f440c003f30 0x7f440c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4426483640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 0x7f440c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4426c84640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f440c004d40 0x7f440c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425c82640 1 -- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 msgr2=0x7f440c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425c82640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4420070500 0x7f440c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425c82640 1 -- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f440c004d40 msgr2=0x7f440c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425c82640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f440c004d40 0x7f440c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4425c82640 1 -- 172.21.15.146:0/1788013566 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c003520 con 0x7f440c003f30 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.766+0000 7f4426c84640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f440c004d40 0x7f440c005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f4425c82640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f440c003f30 0x7f440c004800 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4420074190 tx=0x7f440c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5189549657fcd13d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4410036070 con 0x7f440c003f30 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4410031400 con 0x7f440c003f30 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f4425481640 1 -- 172.21.15.146:0/1788013566 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4410009690 con 0x7f440c003f30 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f44286e7640 1 -- 172.21.15.146:0/1788013566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f440c003f30 msgr2=0x7f440c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f44286e7640 1 --2- 172.21.15.146:0/1788013566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f440c003f30 0x7f440c004800 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4420074190 tx=0x7f440c005830 comp rx=0 tx=0).stop 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.767+0000 7f4426c84640 1 -- 172.21.15.146:0/1788013566 reap_dead start 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.768+0000 7f44286e7640 1 -- 172.21.15.146:0/1788013566 shutdown_connections 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.768+0000 7f44286e7640 1 -- 172.21.15.146:0/1788013566 >> 172.21.15.146:0/1788013566 conn(0x7f44200d1c10 msgr2=0x7f44200733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:30.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.768+0000 7f44286e7640 1 -- 172.21.15.146:0/1788013566 shutdown_connections 2024-04-25T15:22:30.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.768+0000 7f44286e7640 1 -- 172.21.15.146:0/1788013566 wait complete. 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.769+0000 7f44286e7640 1 Processor -- start 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.769+0000 7f44286e7640 1 -- start start 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f44286e7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f440c003f30 0x7f4420151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f44286e7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44200b5950 0x7f44200b5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f44286e7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 0x7f442015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f44286e7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f44200b0e60 con 0x7f442015a4b0 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f44286e7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f44200b0ce0 con 0x7f44200b5950 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f44286e7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f44200b0fe0 con 0x7f440c003f30 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 0x7f442015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 0x7f442015a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50556/0 (socket says 172.21.15.146:50556) 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 -- 172.21.15.146:0/2823644143 learned_addr learned my addr 172.21.15.146:0/2823644143 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426483640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f440c003f30 0x7f4420151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4425c82640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44200b5950 0x7f44200b5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 -- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f440c003f30 msgr2=0x7f4420151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f440c003f30 0x7f4420151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 -- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44200b5950 msgr2=0x7f44200b5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44200b5950 0x7f44200b5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 -- 172.21.15.146:0/2823644143 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442015ae00 con 0x7f442015a4b0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426483640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f440c003f30 0x7f4420151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4425c82640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44200b5950 0x7f44200b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4426c84640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 0x7f442015a8c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4418001350 tx=0x7f441800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.771+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441800eda0 con 0x7f442015a4b0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.772+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4418018070 con 0x7f442015a4b0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.772+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418013660 con 0x7f442015a4b0 2024-04-25T15:22:30.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.772+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44200b44a0 con 0x7f442015a4b0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.772+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44200b4a60 con 0x7f442015a4b0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.773+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442006a0e0 con 0x7f442015a4b0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.773+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4418013800 con 0x7f442015a4b0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.774+0000 7f4416ffd640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f44040745d0 0x7f4404076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.774+0000 7f4426483640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f44040745d0 0x7f4404076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.774+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f4418095920 con 0x7f442015a4b0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.774+0000 7f4426483640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f44040745d0 0x7f4404076ae0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f44200af9f0 tx=0x7f441c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:30.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.778+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f442015ae00 con 0x7f442015a4b0 2024-04-25T15:22:30.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.955+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f44200b6430 con 0x7f442015a4b0 2024-04-25T15:22:30.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.958+0000 7f4416ffd640 1 -- 172.21.15.146:0/2823644143 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v20)=0 v20) ==== 116+0+0 (secure 0 0 0) 0x7f44200b6430 con 0x7f442015a4b0 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f44040745d0 msgr2=0x7f4404076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f44040745d0 0x7f4404076ae0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f44200af9f0 tx=0x7f441c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 msgr2=0x7f442015a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 0x7f442015a8c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4418001350 tx=0x7f441800e530 comp rx=0 tx=0).stop 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 shutdown_connections 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f44040745d0 0x7f4404076ae0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442015a4b0 0x7f442015a8c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44200b5950 0x7f44200b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 --2- 172.21.15.146:0/2823644143 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f440c003f30 0x7f4420151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:30.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 >> 172.21.15.146:0/2823644143 conn(0x7f44200d1c10 msgr2=0x7f44200ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:30.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 shutdown_connections 2024-04-25T15:22:30.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:30.961+0000 7f44286e7640 1 -- 172.21.15.146:0/2823644143 wait complete. 2024-04-25T15:22:30.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_bits 1 2024-04-25T15:22:31.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.083+0000 7f8c959a8640 1 Processor -- start 2024-04-25T15:22:31.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.083+0000 7f8c959a8640 1 -- start start 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c959a8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8c90163c00 0x7f8c9006d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c959a8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8c900d0fb0 0x7f8c900b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c959a8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c9006dea0 con 0x7f8c900bb450 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c959a8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8c900b5230 con 0x7f8c90163c00 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c959a8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8c9006c8c0 con 0x7f8c900d0fb0 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8effd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8c90163c00 0x7f8c9006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8e7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8c900bb450 0x7f8c900d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32806/0 (socket says 172.21.15.146:32806) 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8e7fc640 1 -- 172.21.15.146:0/630487092 learned_addr learned my addr 172.21.15.146:0/630487092 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8f7fe640 1 --2- 172.21.15.146:0/630487092 >> v2:172.21.15.177:3300/0 conn(0x7f8c900d0fb0 0x7f8c900b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8effd640 1 -- 172.21.15.146:0/630487092 >> v2:172.21.15.177:3300/0 conn(0x7f8c900d0fb0 msgr2=0x7f8c900b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8effd640 1 --2- 172.21.15.146:0/630487092 >> v2:172.21.15.177:3300/0 conn(0x7f8c900d0fb0 0x7f8c900b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8effd640 1 -- 172.21.15.146:0/630487092 >> v1:172.21.15.146:6789/0 conn(0x7f8c900bb450 legacy=0x7f8c900d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8effd640 1 -- 172.21.15.146:0/630487092 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9006cba0 con 0x7f8c90163c00 2024-04-25T15:22:31.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.085+0000 7f8c8dffb640 1 -- 172.21.15.146:0/630487092 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348311052 0 0) 0x7f8c9006dea0 con 0x7f8c900bb450 2024-04-25T15:22:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.086+0000 7f8c8effd640 1 --2- 172.21.15.146:0/630487092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c9006d960 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8c900d22c0 tx=0x7f8c8002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2db66ee591b45a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.086+0000 7f8c8dffb640 1 -- 172.21.15.146:0/630487092 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80007740 con 0x7f8c90163c00 2024-04-25T15:22:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.086+0000 7f8c8dffb640 1 -- 172.21.15.146:0/630487092 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c800078e0 con 0x7f8c90163c00 2024-04-25T15:22:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.086+0000 7f8c8dffb640 1 -- 172.21.15.146:0/630487092 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80007cb0 con 0x7f8c90163c00 2024-04-25T15:22:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 -- 172.21.15.146:0/630487092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 msgr2=0x7f8c9006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 --2- 172.21.15.146:0/630487092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c9006d960 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8c900d22c0 tx=0x7f8c8002a5f0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 -- 172.21.15.146:0/630487092 shutdown_connections 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 --2- 172.21.15.146:0/630487092 >> v2:172.21.15.177:3300/0 conn(0x7f8c900d0fb0 0x7f8c900b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 --2- 172.21.15.146:0/630487092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c9006d960 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 -- 172.21.15.146:0/630487092 >> 172.21.15.146:0/630487092 conn(0x7f8c900b7970 msgr2=0x7f8c900b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 -- 172.21.15.146:0/630487092 shutdown_connections 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.087+0000 7f8c959a8640 1 -- 172.21.15.146:0/630487092 wait complete. 2024-04-25T15:22:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.088+0000 7f8c959a8640 1 Processor -- start 2024-04-25T15:22:31.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.089+0000 7f8c959a8640 1 -- start start 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c959a8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c900bb450 0x7f8c900ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c959a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c900d0fb0 0x7f8c901f9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c959a8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c901fa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c959a8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9006c550 con 0x7f8c90163c00 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c959a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9006d160 con 0x7f8c900d0fb0 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c959a8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8c900d13c0 con 0x7f8c900bb450 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c901fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c901fa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50586/0 (socket says 172.21.15.146:50586) 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 -- 172.21.15.146:0/1004164144 learned_addr learned my addr 172.21.15.146:0/1004164144 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8e7fc640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c900d0fb0 0x7f8c901f9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8effd640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c900bb450 0x7f8c900ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 -- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c900bb450 msgr2=0x7f8c900ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c900bb450 0x7f8c900ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 -- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c900d0fb0 msgr2=0x7f8c901f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.090+0000 7f8c8f7fe640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c900d0fb0 0x7f8c901f9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c8f7fe640 1 -- 172.21.15.146:0/1004164144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c90162160 con 0x7f8c90163c00 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c8e7fc640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c900d0fb0 0x7f8c901f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c8f7fe640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c901fa060 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800c550 tx=0x7f8c8800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88015070 con 0x7f8c90163c00 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c88004220 con 0x7f8c90163c00 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88003610 con 0x7f8c90163c00 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c90162440 con 0x7f8c90163c00 2024-04-25T15:22:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.091+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c90162a00 con 0x7f8c90163c00 2024-04-25T15:22:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.092+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c90161ed0 con 0x7f8c90163c00 2024-04-25T15:22:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.093+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f8c880037b0 con 0x7f8c90163c00 2024-04-25T15:22:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.093+0000 7f8c77fff640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8c7c074600 0x7f8c7c076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.093+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f8c880953b0 con 0x7f8c90163c00 2024-04-25T15:22:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.093+0000 7f8c8effd640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8c7c074600 0x7f8c7c076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.094+0000 7f8c8effd640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8c7c074600 0x7f8c7c076b10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8c80034000 tx=0x7f8c80033040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.098+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c8805f7f0 con 0x7f8c90163c00 2024-04-25T15:22:31.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.271+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f8c9006a0e0 con 0x7f8c90163c00 2024-04-25T15:22:31.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.275+0000 7f8c77fff640 1 -- 172.21.15.146:0/1004164144 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v21)=0 v21) ==== 116+0+0 (secure 0 0 0) 0x7f8c8805f0f0 con 0x7f8c90163c00 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8c7c074600 msgr2=0x7f8c7c076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8c7c074600 0x7f8c7c076b10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8c80034000 tx=0x7f8c80033040 comp rx=0 tx=0).stop 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 msgr2=0x7f8c901fa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c901fa060 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800c550 tx=0x7f8c8800ca10 comp rx=0 tx=0).stop 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 shutdown_connections 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8c7c074600 0x7f8c7c076b10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90163c00 0x7f8c901fa060 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c900d0fb0 0x7f8c901f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 --2- 172.21.15.146:0/1004164144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c900bb450 0x7f8c900ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 >> 172.21.15.146:0/1004164144 conn(0x7f8c900b7970 msgr2=0x7f8c900d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:31.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 shutdown_connections 2024-04-25T15:22:31.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.277+0000 7f8c959a8640 1 -- 172.21.15.146:0/1004164144 wait complete. 2024-04-25T15:22:31.290 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:31.290 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:31.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.451+0000 7f690efd6640 1 Processor -- start 2024-04-25T15:22:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.451+0000 7f690efd6640 1 -- start start 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690efd6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6908161980 0x7f6908162590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690efd6640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f69080d1390 0x7f69080d2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690efd6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f69080d31c0 0x7f69080b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690efd6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f69080d27f0 con 0x7f6908161980 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690efd6640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f69080d1be0 con 0x7f69080d1390 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690efd6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f69080d1ec0 con 0x7f69080d31c0 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690cd72640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6908161980 0x7f6908162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690cd72640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6908161980 0x7f6908162590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50592/0 (socket says 172.21.15.146:50592) 2024-04-25T15:22:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690cd72640 1 -- 172.21.15.146:0/3045640268 learned_addr learned my addr 172.21.15.146:0/3045640268 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f6907fff640 1 --2- 172.21.15.146:0/3045640268 >> v2:172.21.15.177:3300/0 conn(0x7f69080d1390 0x7f69080d2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.452+0000 7f690d573640 1 --2- 172.21.15.146:0/3045640268 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f69080d31c0 0x7f69080b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f690cd72640 1 -- 172.21.15.146:0/3045640268 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f69080d31c0 msgr2=0x7f69080b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f690cd72640 1 --2- 172.21.15.146:0/3045640268 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f69080d31c0 0x7f69080b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f690cd72640 1 -- 172.21.15.146:0/3045640268 >> v2:172.21.15.177:3300/0 conn(0x7f69080d1390 msgr2=0x7f69080d2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f690cd72640 1 --2- 172.21.15.146:0/3045640268 >> v2:172.21.15.177:3300/0 conn(0x7f69080d1390 0x7f69080d2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f690cd72640 1 -- 172.21.15.146:0/3045640268 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69080d21a0 con 0x7f6908161980 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f690cd72640 1 --2- 172.21.15.146:0/3045640268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6908161980 0x7f6908162590 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6908074420 tx=0x7f68f802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db96af9f9573a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f69077fe640 1 -- 172.21.15.146:0/3045640268 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8037070 con 0x7f6908161980 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f69077fe640 1 -- 172.21.15.146:0/3045640268 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f8032440 con 0x7f6908161980 2024-04-25T15:22:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.453+0000 7f69077fe640 1 -- 172.21.15.146:0/3045640268 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8032810 con 0x7f6908161980 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 -- 172.21.15.146:0/3045640268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6908161980 msgr2=0x7f6908162590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 --2- 172.21.15.146:0/3045640268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6908161980 0x7f6908162590 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6908074420 tx=0x7f68f802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 -- 172.21.15.146:0/3045640268 shutdown_connections 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 --2- 172.21.15.146:0/3045640268 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f69080d31c0 0x7f69080b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 --2- 172.21.15.146:0/3045640268 >> v2:172.21.15.177:3300/0 conn(0x7f69080d1390 0x7f69080d2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 --2- 172.21.15.146:0/3045640268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6908161980 0x7f6908162590 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 -- 172.21.15.146:0/3045640268 >> 172.21.15.146:0/3045640268 conn(0x7f69080d0240 msgr2=0x7f69080d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 -- 172.21.15.146:0/3045640268 shutdown_connections 2024-04-25T15:22:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.454+0000 7f690efd6640 1 -- 172.21.15.146:0/3045640268 wait complete. 2024-04-25T15:22:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.456+0000 7f690efd6640 1 Processor -- start 2024-04-25T15:22:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.456+0000 7f690efd6640 1 -- start start 2024-04-25T15:22:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.457+0000 7f690efd6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69080d1390 0x7f690815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.457+0000 7f690efd6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 0x7f69080cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.457+0000 7f690efd6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6908161980 0x7f69080b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.457+0000 7f690efd6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f69080d1920 con 0x7f69080d31c0 2024-04-25T15:22:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.457+0000 7f690efd6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69080d17a0 con 0x7f69080d1390 2024-04-25T15:22:31.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.457+0000 7f690efd6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f69080b5780 con 0x7f6908161980 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 0x7f69080cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 0x7f69080cc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50596/0 (socket says 172.21.15.146:50596) 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 -- 172.21.15.146:0/3657315174 learned_addr learned my addr 172.21.15.146:0/3657315174 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f690d573640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6908161980 0x7f69080b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f690cd72640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69080d1390 0x7f690815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 -- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6908161980 msgr2=0x7f69080b8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6908161980 0x7f69080b8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 -- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69080d1390 msgr2=0x7f690815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69080d1390 0x7f690815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 -- 172.21.15.146:0/3657315174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69081f4280 con 0x7f69080d31c0 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f690d573640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6908161980 0x7f69080b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f6907fff640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 0x7f69080cc5b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f68f000d5f0 tx=0x7f68f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00125f0 con 0x7f69080d31c0 2024-04-25T15:22:31.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f0019070 con 0x7f69080d31c0 2024-04-25T15:22:31.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69080cd250 con 0x7f69080d31c0 2024-04-25T15:22:31.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00129d0 con 0x7f69080d31c0 2024-04-25T15:22:31.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.458+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69081f4820 con 0x7f69080d31c0 2024-04-25T15:22:31.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.459+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f690806a0e0 con 0x7f69080d31c0 2024-04-25T15:22:31.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.460+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f68f0012b70 con 0x7f69080d31c0 2024-04-25T15:22:31.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.460+0000 7f69057fa640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68e80745d0 0x7f68e8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.460+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f68f00958d0 con 0x7f69080d31c0 2024-04-25T15:22:31.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.460+0000 7f690cd72640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68e80745d0 0x7f68e8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.461+0000 7f690cd72640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68e80745d0 0x7f68e8076ae0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f68f8034000 tx=0x7f68f802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.465+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68f005fd90 con 0x7f69080d31c0 2024-04-25T15:22:31.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.643+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69080b46f0 con 0x7f69080d31c0 2024-04-25T15:22:31.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.644+0000 7f69057fa640 1 -- 172.21.15.146:0/3657315174 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+17029 (secure 0 0 0) 0x7f68f005f690 con 0x7f69080d31c0 2024-04-25T15:22:31.646 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:31.646 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":56,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:28.267703+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:31.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68e80745d0 msgr2=0x7f68e8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68e80745d0 0x7f68e8076ae0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f68f8034000 tx=0x7f68f802e000 comp rx=0 tx=0).stop 2024-04-25T15:22:31.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 msgr2=0x7f69080cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 0x7f69080cc5b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f68f000d5f0 tx=0x7f68f000dab0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 shutdown_connections 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f68e80745d0 0x7f68e8076ae0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6908161980 0x7f69080b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69080d31c0 0x7f69080cc5b0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 --2- 172.21.15.146:0/3657315174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69080d1390 0x7f690815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.646+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 >> 172.21.15.146:0/3657315174 conn(0x7f69080d0240 msgr2=0x7f69080b6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.647+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 shutdown_connections 2024-04-25T15:22:31.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.647+0000 7f690efd6640 1 -- 172.21.15.146:0/3657315174 wait complete. 2024-04-25T15:22:31.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:31.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.769+0000 7f2521684640 1 Processor -- start 2024-04-25T15:22:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.769+0000 7f2521684640 1 -- start start 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f2521684640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f251c0b4c30 0x7f251c0b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f2521684640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f251c0b9110 0x7f251c1600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f2521684640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c0d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f2521684640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f251c06cc00 con 0x7f251c0b4c30 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f2521684640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f251c0b9520 con 0x7f251c0b9110 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f2521684640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f251c0d1380 con 0x7f251c0d0a30 2024-04-25T15:22:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f251c0b4c30 0x7f251c0b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f251c0b4c30 0x7f251c0b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50610/0 (socket says 172.21.15.146:50610) 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 -- 172.21.15.146:0/775315145 learned_addr learned my addr 172.21.15.146:0/775315145 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251a7fc640 1 --2- 172.21.15.146:0/775315145 >> v2:172.21.15.177:3300/0 conn(0x7f251c0b9110 0x7f251c1600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251b7fe640 1 --2- 172.21.15.146:0/775315145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c0d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 -- 172.21.15.146:0/775315145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 msgr2=0x7f251c0d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 --2- 172.21.15.146:0/775315145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c0d0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 -- 172.21.15.146:0/775315145 >> v2:172.21.15.177:3300/0 conn(0x7f251c0b9110 msgr2=0x7f251c1600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 --2- 172.21.15.146:0/775315145 >> v2:172.21.15.177:3300/0 conn(0x7f251c0b9110 0x7f251c1600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.771+0000 7f251affd640 1 -- 172.21.15.146:0/775315145 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c06c5f0 con 0x7f251c0b4c30 2024-04-25T15:22:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.772+0000 7f251affd640 1 --2- 172.21.15.146:0/775315145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c0b4670 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f251c0b4be0 tx=0x7f250c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a17509d3910151 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.772+0000 7f2519ffb640 1 -- 172.21.15.146:0/775315145 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c036070 con 0x7f251c0b4c30 2024-04-25T15:22:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.772+0000 7f2519ffb640 1 -- 172.21.15.146:0/775315145 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f250c031600 con 0x7f251c0b4c30 2024-04-25T15:22:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.772+0000 7f2519ffb640 1 -- 172.21.15.146:0/775315145 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c0319d0 con 0x7f251c0b4c30 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 -- 172.21.15.146:0/775315145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 msgr2=0x7f251c0b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 --2- 172.21.15.146:0/775315145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c0b4670 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f251c0b4be0 tx=0x7f250c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 -- 172.21.15.146:0/775315145 shutdown_connections 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 --2- 172.21.15.146:0/775315145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c0d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 --2- 172.21.15.146:0/775315145 >> v2:172.21.15.177:3300/0 conn(0x7f251c0b9110 0x7f251c1600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 --2- 172.21.15.146:0/775315145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c0b4670 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 -- 172.21.15.146:0/775315145 >> 172.21.15.146:0/775315145 conn(0x7f251c0b7970 msgr2=0x7f251c0b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 -- 172.21.15.146:0/775315145 shutdown_connections 2024-04-25T15:22:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.773+0000 7f2521684640 1 -- 172.21.15.146:0/775315145 wait complete. 2024-04-25T15:22:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.775+0000 7f2521684640 1 Processor -- start 2024-04-25T15:22:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.775+0000 7f2521684640 1 -- start start 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f2521684640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c1ec0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f2521684640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f251c0b9110 0x7f251c0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f2521684640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c1f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f2521684640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f251c164a90 con 0x7f251c0b4c30 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f2521684640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f251c06c890 con 0x7f251c0b9110 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f2521684640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f251c0d16b0 con 0x7f251c0d0a30 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c1ec0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c1ec0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50618/0 (socket says 172.21.15.146:50618) 2024-04-25T15:22:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 -- 172.21.15.146:0/1350241861 learned_addr learned my addr 172.21.15.146:0/1350241861 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251a7fc640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f251c0b9110 0x7f251c0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251b7fe640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c1f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 -- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 msgr2=0x7f251c1f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c1f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 -- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f251c0b9110 msgr2=0x7f251c0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f251c0b9110 0x7f251c0cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251affd640 1 -- 172.21.15.146:0/1350241861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c0cf750 con 0x7f251c0b4c30 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.776+0000 7f251b7fe640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c1f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.777+0000 7f251affd640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c1ec0f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f250c000f80 tx=0x7f250c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.777+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c036050 con 0x7f251c0b4c30 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.777+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f250c042070 con 0x7f251c0b4c30 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.777+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251c1eb050 con 0x7f251c0b4c30 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.777+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c009510 con 0x7f251c0b4c30 2024-04-25T15:22:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.777+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251c1eb610 con 0x7f251c0b4c30 2024-04-25T15:22:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.778+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f251c06a0e0 con 0x7f251c0b4c30 2024-04-25T15:22:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.778+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f250c007c70 con 0x7f251c0b4c30 2024-04-25T15:22:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.779+0000 7f25037fe640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2508074600 0x7f2508076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.779+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f250c0b6400 con 0x7f251c0b4c30 2024-04-25T15:22:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.779+0000 7f251a7fc640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2508074600 0x7f2508076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.779+0000 7f251a7fc640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2508074600 0x7f2508076b10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f251c0d0f30 tx=0x7f2510008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.784+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f251c0cf750 con 0x7f251c0b4c30 2024-04-25T15:22:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.962+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f251c0d3fa0 con 0x7f251c0b4c30 2024-04-25T15:22:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.962+0000 7f25037fe640 1 -- 172.21.15.146:0/1350241861 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f250c080840 con 0x7f251c0b4c30 2024-04-25T15:22:31.964 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:31.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.964+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2508074600 msgr2=0x7f2508076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.964+0000 7f2521684640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2508074600 0x7f2508076b10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f251c0d0f30 tx=0x7f2510008040 comp rx=0 tx=0).stop 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.964+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 msgr2=0x7f251c1ec0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.964+0000 7f2521684640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c1ec0f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f250c000f80 tx=0x7f250c004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 shutdown_connections 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2508074600 0x7f2508076b10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f251c0d0a30 0x7f251c1f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f251c0b9110 0x7f251c0cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 --2- 172.21.15.146:0/1350241861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f251c0b4c30 0x7f251c1ec0f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 >> 172.21.15.146:0/1350241861 conn(0x7f251c0b7970 msgr2=0x7f251c160ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 shutdown_connections 2024-04-25T15:22:31.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:31.965+0000 7f2521684640 1 -- 172.21.15.146:0/1350241861 wait complete. 2024-04-25T15:22:31.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:32.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.087+0000 7f76a8a92640 1 Processor -- start 2024-04-25T15:22:32.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.087+0000 7f76a8a92640 1 -- start start 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a8a92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76a40b2cd0 0x7f76a40d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a8a92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76a40d2260 con 0x7f76a4071d30 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a8a92640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76a40d2540 con 0x7f76a40d3300 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a8a92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f76a40d2820 con 0x7f76a40b2cd0 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a259c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76a40b2cd0 0x7f76a40d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a2d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f76a4071d30 0x7f76a40cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32808/0 (socket says 172.21.15.146:32808) 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a2d9d640 1 -- 172.21.15.146:0/3238257324 learned_addr learned my addr 172.21.15.146:0/3238257324 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a259c640 1 -- 172.21.15.146:0/3238257324 >> v1:172.21.15.177:6789/0 conn(0x7f76a40d3300 legacy=0x7f76a4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a259c640 1 -- 172.21.15.146:0/3238257324 >> v1:172.21.15.146:6789/0 conn(0x7f76a4071d30 legacy=0x7f76a40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a259c640 1 -- 172.21.15.146:0/3238257324 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a40d2b00 con 0x7f76a40b2cd0 2024-04-25T15:22:32.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.089+0000 7f76a159a640 1 -- 172.21.15.146:0/3238257324 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2267599490 0 0) 0x7f76a40d2260 con 0x7f76a4071d30 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.090+0000 7f76a259c640 1 --2- 172.21.15.146:0/3238257324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40b2cd0 0x7f76a40d1c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7694009de0 tx=0x7f769402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da82a322048ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.090+0000 7f76a159a640 1 -- 172.21.15.146:0/3238257324 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694007740 con 0x7f76a40b2cd0 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.090+0000 7f76a159a640 1 -- 172.21.15.146:0/3238257324 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76940078e0 con 0x7f76a40b2cd0 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.090+0000 7f76a159a640 1 -- 172.21.15.146:0/3238257324 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694007cb0 con 0x7f76a40b2cd0 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 -- 172.21.15.146:0/3238257324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40b2cd0 msgr2=0x7f76a40d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 --2- 172.21.15.146:0/3238257324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40b2cd0 0x7f76a40d1c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7694009de0 tx=0x7f769402a6c0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 -- 172.21.15.146:0/3238257324 shutdown_connections 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 --2- 172.21.15.146:0/3238257324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40b2cd0 0x7f76a40d1c80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 -- 172.21.15.146:0/3238257324 >> 172.21.15.146:0/3238257324 conn(0x7f76a40b52e0 msgr2=0x7f76a40b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:32.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 -- 172.21.15.146:0/3238257324 shutdown_connections 2024-04-25T15:22:32.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.091+0000 7f76a8a92640 1 -- 172.21.15.146:0/3238257324 wait complete. 2024-04-25T15:22:32.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.093+0000 7f76a8a92640 1 Processor -- start 2024-04-25T15:22:32.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.093+0000 7f76a8a92640 1 -- start start 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a8a92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76a40b2cd0 0x7f76a415da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a8a92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 0x7f76a40cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a8a92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76a40b0f60 0x7f76a40b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a8a92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f76a408dbf0 con 0x7f76a40d3300 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a8a92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76a40cfb70 con 0x7f76a40b2cd0 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a8a92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f76a408dd70 con 0x7f76a40b0f60 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 0x7f76a40cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 0x7f76a40cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50638/0 (socket says 172.21.15.146:50638) 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 -- 172.21.15.146:0/729688243 learned_addr learned my addr 172.21.15.146:0/729688243 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a259c640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76a40b2cd0 0x7f76a415da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a2d9d640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76a40b0f60 0x7f76a40b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 -- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76a40b0f60 msgr2=0x7f76a40b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76a40b0f60 0x7f76a40b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 -- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76a40b2cd0 msgr2=0x7f76a415da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76a40b2cd0 0x7f76a415da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.094+0000 7f76a1d9b640 1 -- 172.21.15.146:0/729688243 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a406e120 con 0x7f76a40d3300 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.095+0000 7f76a1d9b640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 0x7f76a40cadb0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f769800c400 tx=0x7f769800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.095+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698017070 con 0x7f76a40d3300 2024-04-25T15:22:32.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.095+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76980052d0 con 0x7f76a40d3300 2024-04-25T15:22:32.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.095+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a406e400 con 0x7f76a40d3300 2024-04-25T15:22:32.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.095+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698003830 con 0x7f76a40d3300 2024-04-25T15:22:32.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.095+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a406e9c0 con 0x7f76a40d3300 2024-04-25T15:22:32.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.096+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a406a0e0 con 0x7f76a40d3300 2024-04-25T15:22:32.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.097+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f769801a070 con 0x7f76a40d3300 2024-04-25T15:22:32.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.097+0000 7f7692ffd640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7684074600 0x7f7684076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.097+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f7698094710 con 0x7f76a40d3300 2024-04-25T15:22:32.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.097+0000 7f76a259c640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7684074600 0x7f7684076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.097+0000 7f76a259c640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7684074600 0x7f7684076b10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7694034000 tx=0x7f769402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.102+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f769805e6a0 con 0x7f76a40d3300 2024-04-25T15:22:32.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.277+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f76a415dfa0 con 0x7f76a40d3300 2024-04-25T15:22:32.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.277+0000 7f7692ffd640 1 -- 172.21.15.146:0/729688243 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f76980629a0 con 0x7f76a40d3300 2024-04-25T15:22:32.280 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7684074600 msgr2=0x7f7684076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7684074600 0x7f7684076b10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7694034000 tx=0x7f769402f000 comp rx=0 tx=0).stop 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 msgr2=0x7f76a40cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 0x7f76a40cadb0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f769800c400 tx=0x7f769800c8c0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 shutdown_connections 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f7684074600 0x7f7684076b10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76a40b0f60 0x7f76a40b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76a40d3300 0x7f76a40cadb0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 --2- 172.21.15.146:0/729688243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76a40b2cd0 0x7f76a415da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 >> 172.21.15.146:0/729688243 conn(0x7f76a40b52e0 msgr2=0x7f76a415efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:32.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 shutdown_connections 2024-04-25T15:22:32.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.279+0000 7f76a8a92640 1 -- 172.21.15.146:0/729688243 wait complete. 2024-04-25T15:22:32.292 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:32.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.401+0000 7f20a2c67640 1 Processor -- start 2024-04-25T15:22:32.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.401+0000 7f20a2c67640 1 -- start start 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.402+0000 7f20a2c67640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f209c09bd80 0x7f209c0ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.402+0000 7f20a2c67640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f209c0d2680 0x7f209c0d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.402+0000 7f20a2c67640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f209c0d2fd0 con 0x7f209c15e070 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a2c67640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f209c0ce920 con 0x7f209c0d2680 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a2c67640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f209c0cec00 con 0x7f209c09bd80 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f209c0d2680 0x7f209c0d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f209c0d2680 0x7f209c0d2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50642/0 (socket says 172.21.15.146:50642) 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 -- 172.21.15.146:0/601545545 learned_addr learned my addr 172.21.15.146:0/601545545 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f209bfff640 1 --2- 172.21.15.146:0/601545545 >> v2:172.21.15.177:3300/0 conn(0x7f209c09bd80 0x7f209c0ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 -- 172.21.15.146:0/601545545 >> v2:172.21.15.177:3300/0 conn(0x7f209c09bd80 msgr2=0x7f209c0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 --2- 172.21.15.146:0/601545545 >> v2:172.21.15.177:3300/0 conn(0x7f209c09bd80 0x7f209c0ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 -- 172.21.15.146:0/601545545 >> v1:172.21.15.177:6789/0 conn(0x7f209c15e070 legacy=0x7f209c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.403+0000 7f20a1204640 1 -- 172.21.15.146:0/601545545 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209c0ceee0 con 0x7f209c0d2680 2024-04-25T15:22:32.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.404+0000 7f20a1204640 1 --2- 172.21.15.146:0/601545545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c0d2680 0x7f209c0d2a90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f209c073720 tx=0x7f209402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe77b641d0e24c3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.404+0000 7f209b7fe640 1 -- 172.21.15.146:0/601545545 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094036070 con 0x7f209c0d2680 2024-04-25T15:22:32.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.404+0000 7f209b7fe640 1 -- 172.21.15.146:0/601545545 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2094031600 con 0x7f209c0d2680 2024-04-25T15:22:32.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.404+0000 7f209b7fe640 1 -- 172.21.15.146:0/601545545 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20940319d0 con 0x7f209c0d2680 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 -- 172.21.15.146:0/601545545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c0d2680 msgr2=0x7f209c0d2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 --2- 172.21.15.146:0/601545545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c0d2680 0x7f209c0d2a90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f209c073720 tx=0x7f209402a730 comp rx=0 tx=0).stop 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 -- 172.21.15.146:0/601545545 shutdown_connections 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 --2- 172.21.15.146:0/601545545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c0d2680 0x7f209c0d2a90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 --2- 172.21.15.146:0/601545545 >> v2:172.21.15.177:3300/0 conn(0x7f209c09bd80 0x7f209c0ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 -- 172.21.15.146:0/601545545 >> 172.21.15.146:0/601545545 conn(0x7f209c0b7400 msgr2=0x7f209c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 -- 172.21.15.146:0/601545545 shutdown_connections 2024-04-25T15:22:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.405+0000 7f20a2c67640 1 -- 172.21.15.146:0/601545545 wait complete. 2024-04-25T15:22:32.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.407+0000 7f20a2c67640 1 Processor -- start 2024-04-25T15:22:32.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.407+0000 7f20a2c67640 1 -- start start 2024-04-25T15:22:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a2c67640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f209c09bd80 0x7f209c1602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a2c67640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f209c0d2680 0x7f209c0b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a2c67640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 0x7f209c0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a2c67640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f209c0ce5a0 con 0x7f209c15e070 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a2c67640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f209c0cf4a0 con 0x7f209c0d2680 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a2c67640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f209c06cde0 con 0x7f209c09bd80 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a1204640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 0x7f209c0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a1204640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 0x7f209c0c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50654/0 (socket says 172.21.15.146:50654) 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a1204640 1 -- 172.21.15.146:0/22832107 learned_addr learned my addr 172.21.15.146:0/22832107 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f209bfff640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f209c0d2680 0x7f209c0b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.408+0000 7f20a0a03640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f209c09bd80 0x7f209c1602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a1204640 1 -- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f209c09bd80 msgr2=0x7f209c1602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a1204640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f209c09bd80 0x7f209c1602f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a1204640 1 -- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f209c0d2680 msgr2=0x7f209c0b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a1204640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f209c0d2680 0x7f209c0b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a1204640 1 -- 172.21.15.146:0/22832107 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209c160bf0 con 0x7f209c15e070 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a1204640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 0x7f209c0c83f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2094009f40 tx=0x7f2094004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209403f020 con 0x7f209c15e070 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2094037040 con 0x7f209c15e070 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209c160ed0 con 0x7f209c15e070 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094036070 con 0x7f209c15e070 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.409+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f209c0b2230 con 0x7f209c15e070 2024-04-25T15:22:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.410+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f209c160960 con 0x7f209c15e070 2024-04-25T15:22:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.411+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f2094042020 con 0x7f209c15e070 2024-04-25T15:22:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.411+0000 7f20997fa640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20880745d0 0x7f2088076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.411+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f20940b5c10 con 0x7f209c15e070 2024-04-25T15:22:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.411+0000 7f20a0a03640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20880745d0 0x7f2088076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.412+0000 7f20a0a03640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20880745d0 0x7f2088076ae0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f208c007fd0 tx=0x7f208c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.416+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2094049020 con 0x7f209c15e070 2024-04-25T15:22:32.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.592+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f209c1613f0 con 0x7f209c15e070 2024-04-25T15:22:32.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.593+0000 7f20997fa640 1 -- 172.21.15.146:0/22832107 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f209407fa00 con 0x7f209c15e070 2024-04-25T15:22:32.595 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20880745d0 msgr2=0x7f2088076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20880745d0 0x7f2088076ae0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f208c007fd0 tx=0x7f208c009040 comp rx=0 tx=0).stop 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 msgr2=0x7f209c0c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 0x7f209c0c83f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2094009f40 tx=0x7f2094004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 shutdown_connections 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20880745d0 0x7f2088076ae0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f209c15e070 0x7f209c0c83f0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f209c0d2680 0x7f209c0b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 --2- 172.21.15.146:0/22832107 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f209c09bd80 0x7f209c1602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 >> 172.21.15.146:0/22832107 conn(0x7f209c0b7400 msgr2=0x7f209c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 shutdown_connections 2024-04-25T15:22:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.595+0000 7f20a2c67640 1 -- 172.21.15.146:0/22832107 wait complete. 2024-04-25T15:22:32.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:32.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.716+0000 7f870678f640 1 Processor -- start 2024-04-25T15:22:32.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.716+0000 7f870678f640 1 -- start start 2024-04-25T15:22:32.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f870678f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 0x7f87000d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f870678f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f870006ee60 0x7f870008cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f870678f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870008d5b0 con 0x7f870008d1a0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f870678f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f8700160680 con 0x7f870006ee60 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f870678f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f87000d20e0 con 0x7f87000b2df0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f8704d2c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f870008d1a0 0x7f87000cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32810/0 (socket says 172.21.15.146:32810) 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f8704d2c640 1 -- 172.21.15.146:0/114335762 learned_addr learned my addr 172.21.15.146:0/114335762 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86fffff640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 0x7f87000d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 77791987 0 0) 0x7f86f8001490 con 0x7f870008d1a0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/114335762 >> v2:172.21.15.177:3300/0 conn(0x7f870006ee60 0x7f870008cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e8003520 con 0x7f870008d1a0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3442784417 0 0) 0x7f86e8003520 con 0x7f870008d1a0 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 msgr2=0x7f87000d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 0x7f87000d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 >> v2:172.21.15.177:3300/0 conn(0x7f870006ee60 msgr2=0x7f870008cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:32.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 --2- 172.21.15.146:0/114335762 >> v2:172.21.15.177:3300/0 conn(0x7f870006ee60 0x7f870008cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87000d23c0 con 0x7f870008d1a0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f86f8001bc0 con 0x7f870008d1a0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.718+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 >> v1:172.21.15.146:6789/0 conn(0x7f870008d1a0 legacy=0x7f87000cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86feffd640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e80039b0 0x7f86e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86feffd640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86e8004800 0x7f86e8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86feffd640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 0x7f86e8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8700160680 con 0x7f86e8004800 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f87000d20e0 con 0x7f86e80039b0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f86e800ba70 con 0x7f87000b2df0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86e8004800 0x7f86e8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86fffff640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e80039b0 0x7f86e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f8704d2c640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 0x7f86e8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86ff7fe640 1 -- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 msgr2=0x7f86e8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:32.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87000b2df0 0x7f86e8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86ff7fe640 1 -- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e80039b0 msgr2=0x7f86e8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e80039b0 0x7f86e8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86ff7fe640 1 -- 172.21.15.146:0/114335762 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e8003520 con 0x7f86e8004800 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.719+0000 7f86fffff640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e80039b0 0x7f86e8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86e8004800 0x7f86e8004c10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009ac0 tx=0x7f86e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57afd79e5457a9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4036070 con 0x7f86e8004800 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f4031400 con 0x7f86e8004800 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f86feffd640 1 -- 172.21.15.146:0/114335762 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4009690 con 0x7f86e8004800 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f870678f640 1 -- 172.21.15.146:0/114335762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86e8004800 msgr2=0x7f86e8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f870678f640 1 --2- 172.21.15.146:0/114335762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86e8004800 0x7f86e8004c10 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009ac0 tx=0x7f86e8005830 comp rx=0 tx=0).stop 2024-04-25T15:22:32.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f8704d2c640 1 -- 172.21.15.146:0/114335762 reap_dead start 2024-04-25T15:22:32.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f870678f640 1 -- 172.21.15.146:0/114335762 shutdown_connections 2024-04-25T15:22:32.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f870678f640 1 -- 172.21.15.146:0/114335762 >> 172.21.15.146:0/114335762 conn(0x7f87000b5360 msgr2=0x7f87000b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:32.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f870678f640 1 -- 172.21.15.146:0/114335762 shutdown_connections 2024-04-25T15:22:32.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.720+0000 7f870678f640 1 -- 172.21.15.146:0/114335762 wait complete. 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.722+0000 7f870678f640 1 Processor -- start 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.722+0000 7f870678f640 1 -- start start 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e8004800 0x7f870014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8700161140 0x7f8700161550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 0x7f87000b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f87000d2c60 con 0x7f87000b1020 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f87000d2980 con 0x7f86e8004800 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f87000b91e0 con 0x7f8700161140 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 0x7f87000b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 0x7f87000b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50672/0 (socket says 172.21.15.146:50672) 2024-04-25T15:22:32.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 -- 172.21.15.146:0/1087481492 learned_addr learned my addr 172.21.15.146:0/1087481492 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8700161140 0x7f8700161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86fffff640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e8004800 0x7f870014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 -- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8700161140 msgr2=0x7f8700161550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8700161140 0x7f8700161550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 -- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e8004800 msgr2=0x7f870014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e8004800 0x7f870014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 -- 172.21.15.146:0/1087481492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87000b1970 con 0x7f87000b1020 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86ff7fe640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8700161140 0x7f8700161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86fffff640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e8004800 0x7f870014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f8704d2c640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 0x7f87000b1430 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f86f80013d0 tx=0x7f86f8007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f800e960 con 0x7f87000b1020 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f800eb40 con 0x7f87000b1020 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87001f62d0 con 0x7f87000b1020 2024-04-25T15:22:32.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f800ed30 con 0x7f87000b1020 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.724+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87000c9ad0 con 0x7f87000b1020 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.725+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87000c9350 con 0x7f87000b1020 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.726+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f86f801b020 con 0x7f87000b1020 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.726+0000 7f86fcff9640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f86ec074600 0x7f86ec076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.727+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f86f8094f50 con 0x7f87000b1020 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.727+0000 7f86fffff640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f86ec074600 0x7f86ec076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:32.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.727+0000 7f86fffff640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f86ec074600 0x7f86ec076b10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f86f0006fd0 tx=0x7f86f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:32.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.731+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87000c9350 con 0x7f87000b1020 2024-04-25T15:22:32.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.916+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f870006a0e0 con 0x7f87000b1020 2024-04-25T15:22:32.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.917+0000 7f86fcff9640 1 -- 172.21.15.146:0/1087481492 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f87000b1970 con 0x7f87000b1020 2024-04-25T15:22:32.919 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:22:32.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f86ec074600 msgr2=0x7f86ec076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f86ec074600 0x7f86ec076b10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f86f0006fd0 tx=0x7f86f0008040 comp rx=0 tx=0).stop 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 msgr2=0x7f87000b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 0x7f87000b1430 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f86f80013d0 tx=0x7f86f8007b10 comp rx=0 tx=0).stop 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 shutdown_connections 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f86ec074600 0x7f86ec076b10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87000b1020 0x7f87000b1430 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8700161140 0x7f8700161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 --2- 172.21.15.146:0/1087481492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86e8004800 0x7f870014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 >> 172.21.15.146:0/1087481492 conn(0x7f87000b5360 msgr2=0x7f87000d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 shutdown_connections 2024-04-25T15:22:32.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:32.919+0000 7f870678f640 1 -- 172.21.15.146:0/1087481492 wait complete. 2024-04-25T15:22:32.932 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:32.932 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:32.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:33.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.093+0000 7f01f6030640 1 Processor -- start 2024-04-25T15:22:33.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.093+0000 7f01f6030640 1 -- start start 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01f6030640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f01f00cbc90 0x7f01f0153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01f6030640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01f6030640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f00cf420 con 0x7f01f00d0ac0 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01f6030640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f01f01541e0 con 0x7f01f00cbc90 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01f6030640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f01f00cfd10 con 0x7f01f00d01a0 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01eeffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f01f00cbc90 0x7f01f0153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f01f00cbc90 0x7f01f0153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56200/0 (socket says 172.21.15.146:56200) 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01effff640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f01f00d0ac0 0x7f01f00d0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:49416/0 (socket says 172.21.15.146:49416) 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01eeffd640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f015dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:38540/0 (socket says 172.21.15.146:38540) 2024-04-25T15:22:33.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 -- 172.21.15.146:0/2135012002 learned_addr learned my addr 172.21.15.146:0/2135012002 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:33.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 -- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 msgr2=0x7f01f015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:33.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 --2- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f015dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 -- 172.21.15.146:0/2135012002 >> v1:172.21.15.177:6789/0 conn(0x7f01f00d0ac0 legacy=0x7f01f00d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ef7fe640 1 -- 172.21.15.146:0/2135012002 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f00cc270 con 0x7f01f00cbc90 2024-04-25T15:22:33.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01ee7fc640 1 -- 172.21.15.146:0/2135012002 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208383347 0 0) 0x7f01f00cf420 con 0x7f01f00d0ac0 2024-04-25T15:22:33.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.095+0000 7f01eeffd640 1 --2- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:33.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.096+0000 7f01ef7fe640 1 --2- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00cbc90 0x7f01f0153ca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009870 tx=0x7f01e002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e4385a7acef4d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:33.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.096+0000 7f01ee7fc640 1 -- 172.21.15.146:0/2135012002 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0037070 con 0x7f01f00cbc90 2024-04-25T15:22:33.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.096+0000 7f01ee7fc640 1 -- 172.21.15.146:0/2135012002 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e0032440 con 0x7f01f00cbc90 2024-04-25T15:22:33.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.096+0000 7f01ee7fc640 1 -- 172.21.15.146:0/2135012002 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0032810 con 0x7f01f00cbc90 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 -- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00cbc90 msgr2=0x7f01f0153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 --2- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00cbc90 0x7f01f0153ca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009870 tx=0x7f01e002a270 comp rx=0 tx=0).stop 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 -- 172.21.15.146:0/2135012002 shutdown_connections 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 --2- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 --2- 172.21.15.146:0/2135012002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00cbc90 0x7f01f0153ca0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 -- 172.21.15.146:0/2135012002 >> 172.21.15.146:0/2135012002 conn(0x7f01f00ce0c0 msgr2=0x7f01f00ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 -- 172.21.15.146:0/2135012002 shutdown_connections 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.097+0000 7f01f6030640 1 -- 172.21.15.146:0/2135012002 wait complete. 2024-04-25T15:22:33.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.099+0000 7f01f6030640 1 Processor -- start 2024-04-25T15:22:33.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.099+0000 7f01f6030640 1 -- start start 2024-04-25T15:22:33.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01f6030640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 0x7f01f01f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01f6030640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f00b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01f6030640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00d0ac0 0x7f01f0159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01f6030640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f01f015e1a0 con 0x7f01f00cbc90 2024-04-25T15:22:33.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01f6030640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01f00cf9a0 con 0x7f01f00d0ac0 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01f6030640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f01f00d05b0 con 0x7f01f00d01a0 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01ef7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 0x7f01f01f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01ef7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 0x7f01f01f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50676/0 (socket says 172.21.15.146:50676) 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01ef7fe640 1 -- 172.21.15.146:0/751220668 learned_addr learned my addr 172.21.15.146:0/751220668 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01eeffd640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f00b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.100+0000 7f01effff640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00d0ac0 0x7f01f0159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01ef7fe640 1 -- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 msgr2=0x7f01f00b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01ef7fe640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f00b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01ef7fe640 1 -- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00d0ac0 msgr2=0x7f01f0159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01ef7fe640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00d0ac0 0x7f01f0159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01ef7fe640 1 -- 172.21.15.146:0/751220668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f014edc0 con 0x7f01f00cbc90 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01ef7fe640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 0x7f01f01f3910 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f01e0035040 tx=0x7f01e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0037040 con 0x7f01f00cbc90 2024-04-25T15:22:33.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e0004030 con 0x7f01f00cbc90 2024-04-25T15:22:33.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f014f0a0 con 0x7f01f00cbc90 2024-04-25T15:22:33.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0004420 con 0x7f01f00cbc90 2024-04-25T15:22:33.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.101+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f014f660 con 0x7f01f00cbc90 2024-04-25T15:22:33.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.102+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f006a0e0 con 0x7f01f00cbc90 2024-04-25T15:22:33.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.103+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f01e003a070 con 0x7f01f00cbc90 2024-04-25T15:22:33.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.103+0000 7f01d7fff640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f01dc0745d0 0x7f01dc076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.103+0000 7f01eeffd640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f01dc0745d0 0x7f01dc076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.103+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f01e00b5a70 con 0x7f01f00cbc90 2024-04-25T15:22:33.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.104+0000 7f01eeffd640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f01dc0745d0 0x7f01dc076ae0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f01f015dcd0 tx=0x7f01e4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:33.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.108+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01e0041020 con 0x7f01f00cbc90 2024-04-25T15:22:33.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.284+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01f00711c0 con 0x7f01f00cbc90 2024-04-25T15:22:33.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.285+0000 7f01d7fff640 1 -- 172.21.15.146:0/751220668 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+17029 (secure 0 0 0) 0x7f01e007f8e0 con 0x7f01f00cbc90 2024-04-25T15:22:33.288 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:33.288 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":56,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:28.267703+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:33.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.287+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f01dc0745d0 msgr2=0x7f01dc076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.287+0000 7f01f6030640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f01dc0745d0 0x7f01dc076ae0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f01f015dcd0 tx=0x7f01e4009190 comp rx=0 tx=0).stop 2024-04-25T15:22:33.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.287+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 msgr2=0x7f01f01f3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.287+0000 7f01f6030640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 0x7f01f01f3910 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f01e0035040 tx=0x7f01e00046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 shutdown_connections 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f01dc0745d0 0x7f01dc076ae0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01f00d0ac0 0x7f01f0159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01f00d01a0 0x7f01f00b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 --2- 172.21.15.146:0/751220668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01f00cbc90 0x7f01f01f3910 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 >> 172.21.15.146:0/751220668 conn(0x7f01f00ce0c0 msgr2=0x7f01f015e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 shutdown_connections 2024-04-25T15:22:33.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.288+0000 7f01f6030640 1 -- 172.21.15.146:0/751220668 wait complete. 2024-04-25T15:22:33.301 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:33.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:33.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.411+0000 7fb504ab6640 1 Processor -- start 2024-04-25T15:22:33.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.411+0000 7fb504ab6640 1 -- start start 2024-04-25T15:22:33.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb504ab6640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb5000ccd40 0x7fb50009bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb504ab6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb5000b2400 0x7fb5000b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb504ab6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5000b1960 con 0x7fb5000d2680 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb504ab6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb5000b3ce0 con 0x7fb5000b2400 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb504ab6640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb5000b1c00 con 0x7fb5000ccd40 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fed9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb5000b2400 0x7fb5000b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fdd9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb5000d2680 0x7fb5000d2a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32818/0 (socket says 172.21.15.146:32818) 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fdd9b640 1 -- 172.21.15.146:0/3803667290 learned_addr learned my addr 172.21.15.146:0/3803667290 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fed9d640 1 -- 172.21.15.146:0/3803667290 >> v2:172.21.15.177:3300/0 conn(0x7fb5000ccd40 msgr2=0x7fb50009bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fe59c640 1 --2- 172.21.15.146:0/3803667290 >> v2:172.21.15.177:3300/0 conn(0x7fb5000ccd40 0x7fb50009bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fed9d640 1 --2- 172.21.15.146:0/3803667290 >> v2:172.21.15.177:3300/0 conn(0x7fb5000ccd40 0x7fb50009bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fd59a640 1 -- 172.21.15.146:0/3803667290 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228865143 0 0) 0x7fb5000b1960 con 0x7fb5000d2680 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fed9d640 1 -- 172.21.15.146:0/3803667290 >> v1:172.21.15.146:6789/0 conn(0x7fb5000d2680 legacy=0x7fb5000d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fed9d640 1 -- 172.21.15.146:0/3803667290 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5000b1ee0 con 0x7fb5000b2400 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.413+0000 7fb4fe59c640 1 --2- 172.21.15.146:0/3803667290 >> v2:172.21.15.177:3300/0 conn(0x7fb5000ccd40 0x7fb50009bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:33.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.414+0000 7fb4fed9d640 1 --2- 172.21.15.146:0/3803667290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000b2400 0x7fb5000b2810 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb5000d2650 tx=0x7fb4f802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bae3b9ec8e0492 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:33.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.414+0000 7fb4fd59a640 1 -- 172.21.15.146:0/3803667290 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8037070 con 0x7fb5000b2400 2024-04-25T15:22:33.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.414+0000 7fb4fd59a640 1 -- 172.21.15.146:0/3803667290 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f8032440 con 0x7fb5000b2400 2024-04-25T15:22:33.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.414+0000 7fb4fd59a640 1 -- 172.21.15.146:0/3803667290 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8032810 con 0x7fb5000b2400 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 -- 172.21.15.146:0/3803667290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000b2400 msgr2=0x7fb5000b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 --2- 172.21.15.146:0/3803667290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000b2400 0x7fb5000b2810 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb5000d2650 tx=0x7fb4f802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 -- 172.21.15.146:0/3803667290 shutdown_connections 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 --2- 172.21.15.146:0/3803667290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000b2400 0x7fb5000b2810 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 --2- 172.21.15.146:0/3803667290 >> v2:172.21.15.177:3300/0 conn(0x7fb5000ccd40 0x7fb50009bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 -- 172.21.15.146:0/3803667290 >> 172.21.15.146:0/3803667290 conn(0x7fb5000b7400 msgr2=0x7fb5000b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 -- 172.21.15.146:0/3803667290 shutdown_connections 2024-04-25T15:22:33.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.415+0000 7fb504ab6640 1 -- 172.21.15.146:0/3803667290 wait complete. 2024-04-25T15:22:33.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.417+0000 7fb504ab6640 1 Processor -- start 2024-04-25T15:22:33.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.417+0000 7fb504ab6640 1 -- start start 2024-04-25T15:22:33.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb504ab6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5000b2400 0x7fb500153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb504ab6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5000ccd40 0x7fb5000ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb504ab6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 0x7fb50009e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb504ab6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb5000b3ac0 con 0x7fb5000d2680 2024-04-25T15:22:33.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb504ab6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5000b3940 con 0x7fb5000b2400 2024-04-25T15:22:33.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb504ab6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb50009b440 con 0x7fb5000ccd40 2024-04-25T15:22:33.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb4fed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 0x7fb50009e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb4fed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 0x7fb50009e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50698/0 (socket says 172.21.15.146:50698) 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb4fed9d640 1 -- 172.21.15.146:0/4115315090 learned_addr learned my addr 172.21.15.146:0/4115315090 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb4fdd9b640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5000ccd40 0x7fb5000ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.418+0000 7fb4fe59c640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5000b2400 0x7fb500153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4fed9d640 1 -- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5000ccd40 msgr2=0x7fb5000ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4fed9d640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5000ccd40 0x7fb5000ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4fed9d640 1 -- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5000b2400 msgr2=0x7fb500153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4fed9d640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5000b2400 0x7fb500153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4fed9d640 1 -- 172.21.15.146:0/4115315090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5001541e0 con 0x7fb5000d2680 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4fed9d640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 0x7fb50009e150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009890 tx=0x7fb4f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8037040 con 0x7fb5000d2680 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f8004230 con 0x7fb5000d2680 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8032e20 con 0x7fb5000d2680 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5000cb650 con 0x7fb5000d2680 2024-04-25T15:22:33.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.419+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5000cbc10 con 0x7fb5000d2680 2024-04-25T15:22:33.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.420+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50006a0e0 con 0x7fb5000d2680 2024-04-25T15:22:33.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.421+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fb4f8030770 con 0x7fb5000d2680 2024-04-25T15:22:33.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.421+0000 7fb4eeffd640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4e0074600 0x7fb4e0076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:33.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.421+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fb4f80b6680 con 0x7fb5000d2680 2024-04-25T15:22:33.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.421+0000 7fb4fe59c640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4e0074600 0x7fb4e0076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:33.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.422+0000 7fb4fe59c640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4e0074600 0x7fb4e0076b10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb5000b8040 tx=0x7fb4f0007810 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:33.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.426+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb4f80804f0 con 0x7fb5000d2680 2024-04-25T15:22:33.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.599+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb5000cb3e0 con 0x7fb5000d2680 2024-04-25T15:22:33.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.600+0000 7fb4eeffd640 1 -- 172.21.15.146:0/4115315090 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fb4f80847f0 con 0x7fb5000d2680 2024-04-25T15:22:33.601 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:33.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.601+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4e0074600 msgr2=0x7fb4e0076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4e0074600 0x7fb4e0076b10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb5000b8040 tx=0x7fb4f0007810 comp rx=0 tx=0).stop 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 msgr2=0x7fb50009e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 0x7fb50009e150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009890 tx=0x7fb4f80046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 shutdown_connections 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb4e0074600 0x7fb4e0076b10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5000d2680 0x7fb50009e150 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5000ccd40 0x7fb5000ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 --2- 172.21.15.146:0/4115315090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5000b2400 0x7fb500153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 >> 172.21.15.146:0/4115315090 conn(0x7fb5000b7400 msgr2=0x7fb5000b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 shutdown_connections 2024-04-25T15:22:33.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.602+0000 7fb504ab6640 1 -- 172.21.15.146:0/4115315090 wait complete. 2024-04-25T15:22:33.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:33.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.724+0000 7f9ab8585640 1 Processor -- start 2024-04-25T15:22:34.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.724+0000 7f9ab8585640 1 -- start start 2024-04-25T15:22:34.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab8585640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9ab00d0fb0 0x7f9ab00b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab8585640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab00b5230 con 0x7f9ab00bb450 2024-04-25T15:22:34.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab8585640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab006c8c0 con 0x7f9ab0163c00 2024-04-25T15:22:34.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab8585640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab006cba0 con 0x7f9ab00d0fb0 2024-04-25T15:22:34.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab6b22640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9ab00d0fb0 0x7f9ab00b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab5b20640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9ab00bb450 0x7f9ab00d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32822/0 (socket says 172.21.15.146:32822) 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.725+0000 7f9ab5b20640 1 -- 172.21.15.146:0/4019851473 learned_addr learned my addr 172.21.15.146:0/4019851473 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.726+0000 7f9ab6b22640 1 -- 172.21.15.146:0/4019851473 >> v1:172.21.15.177:6789/0 conn(0x7f9ab0163c00 legacy=0x7f9ab006d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.726+0000 7f9ab531f640 1 -- 172.21.15.146:0/4019851473 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856521134 0 0) 0x7f9ab00b5230 con 0x7f9ab00bb450 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.726+0000 7f9ab6b22640 1 -- 172.21.15.146:0/4019851473 >> v1:172.21.15.146:6789/0 conn(0x7f9ab00bb450 legacy=0x7f9ab00d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.726+0000 7f9ab6b22640 1 -- 172.21.15.146:0/4019851473 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab006ce80 con 0x7f9ab00d0fb0 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.726+0000 7f9ab6b22640 1 --2- 172.21.15.146:0/4019851473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00d0fb0 0x7f9ab00b8fb0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80097a0 tx=0x7f9aa802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a783f2abf154e2bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab531f640 1 -- 172.21.15.146:0/4019851473 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8037070 con 0x7f9ab00d0fb0 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab531f640 1 -- 172.21.15.146:0/4019851473 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa8032440 con 0x7f9ab00d0fb0 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab531f640 1 -- 172.21.15.146:0/4019851473 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8032810 con 0x7f9ab00d0fb0 2024-04-25T15:22:34.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 -- 172.21.15.146:0/4019851473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00d0fb0 msgr2=0x7f9ab00b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 --2- 172.21.15.146:0/4019851473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00d0fb0 0x7f9ab00b8fb0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80097a0 tx=0x7f9aa802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 -- 172.21.15.146:0/4019851473 shutdown_connections 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 --2- 172.21.15.146:0/4019851473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00d0fb0 0x7f9ab00b8fb0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 -- 172.21.15.146:0/4019851473 >> 172.21.15.146:0/4019851473 conn(0x7f9ab00b7970 msgr2=0x7f9ab00b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 -- 172.21.15.146:0/4019851473 shutdown_connections 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.727+0000 7f9ab8585640 1 -- 172.21.15.146:0/4019851473 wait complete. 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.729+0000 7f9ab8585640 1 Processor -- start 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.729+0000 7f9ab8585640 1 -- start start 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.730+0000 7f9ab8585640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 0x7f9ab00cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.730+0000 7f9ab8585640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ab00d0fb0 0x7f9ab01f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.730+0000 7f9ab8585640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0163c00 0x7f9ab0154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.730+0000 7f9ab8585640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab00d13c0 con 0x7f9ab00bb450 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.730+0000 7f9ab8585640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab006c550 con 0x7f9ab0163c00 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.730+0000 7f9ab8585640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab00d1540 con 0x7f9ab00d0fb0 2024-04-25T15:22:34.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 0x7f9ab00cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 0x7f9ab00cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50726/0 (socket says 172.21.15.146:50726) 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 -- 172.21.15.146:0/2502952212 learned_addr learned my addr 172.21.15.146:0/2502952212 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6b22640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0163c00 0x7f9ab0154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab5b20640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ab00d0fb0 0x7f9ab01f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 -- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ab00d0fb0 msgr2=0x7f9ab01f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ab00d0fb0 0x7f9ab01f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 -- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0163c00 msgr2=0x7f9ab0154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0163c00 0x7f9ab0154790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 -- 172.21.15.146:0/2502952212 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab00ce2a0 con 0x7f9ab00bb450 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab5b20640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ab00d0fb0 0x7f9ab01f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab6321640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 0x7f9ab00cdd60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c730 tx=0x7f9aac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac003b70 con 0x7f9ab00bb450 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aac003d30 con 0x7f9ab00bb450 2024-04-25T15:22:34.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab00ce7b0 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac00aa90 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.731+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab00ced70 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.732+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab006a0e0 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.733+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f9aac00ac30 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.733+0000 7f9aa6ffd640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a940745d0 0x7f9a94076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.733+0000 7f9ab5b20640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a940745d0 0x7f9a94076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.733+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f9aac095350 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.734+0000 7f9ab5b20640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a940745d0 0x7f9a94076ae0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0007fd0 tx=0x7f9aa0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.738+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9aac05f8a0 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.912+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9ab00d3ed0 con 0x7f9ab00bb450 2024-04-25T15:22:34.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.912+0000 7f9aa6ffd640 1 -- 172.21.15.146:0/2502952212 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f9aac05f1a0 con 0x7f9ab00bb450 2024-04-25T15:22:34.267 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a940745d0 msgr2=0x7f9a94076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a940745d0 0x7f9a94076ae0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0007fd0 tx=0x7f9aa0007de0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 msgr2=0x7f9ab00cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 0x7f9ab00cdd60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c730 tx=0x7f9aac00cbf0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 shutdown_connections 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a940745d0 0x7f9a94076ae0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0163c00 0x7f9ab0154790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ab00d0fb0 0x7f9ab01f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 --2- 172.21.15.146:0/2502952212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab00bb450 0x7f9ab00cdd60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 >> 172.21.15.146:0/2502952212 conn(0x7f9ab00b7970 msgr2=0x7f9ab00ba300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:34.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 shutdown_connections 2024-04-25T15:22:34.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:33.914+0000 7f9ab8585640 1 -- 172.21.15.146:0/2502952212 wait complete. 2024-04-25T15:22:34.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:34.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.382+0000 7f3a3ca92640 1 Processor -- start 2024-04-25T15:22:34.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.382+0000 7f3a3ca92640 1 -- start start 2024-04-25T15:22:34.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3ca92640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3a380741f0 0x7f3a380b4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3ca92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3a380ae660 0x7f3a380b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3ca92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a3815d940 con 0x7f3a3815d480 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3ca92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3806fe30 con 0x7f3a380ae660 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3ca92640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3a380702c0 con 0x7f3a380741f0 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a36d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3a380ae660 0x7f3a380b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3a3815d480 0x7f3a38073ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32838/0 (socket says 172.21.15.146:32838) 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3659c640 1 -- 172.21.15.146:0/979683322 learned_addr learned my addr 172.21.15.146:0/979683322 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a35d9b640 1 --2- 172.21.15.146:0/979683322 >> v2:172.21.15.177:3300/0 conn(0x7f3a380741f0 0x7f3a380b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a36d9d640 1 -- 172.21.15.146:0/979683322 >> v2:172.21.15.177:3300/0 conn(0x7f3a380741f0 msgr2=0x7f3a380b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a36d9d640 1 --2- 172.21.15.146:0/979683322 >> v2:172.21.15.177:3300/0 conn(0x7f3a380741f0 0x7f3a380b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a36d9d640 1 -- 172.21.15.146:0/979683322 >> v1:172.21.15.146:6789/0 conn(0x7f3a3815d480 legacy=0x7f3a38073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a3559a640 1 -- 172.21.15.146:0/979683322 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987396925 0 0) 0x7f3a3815d940 con 0x7f3a3815d480 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.384+0000 7f3a36d9d640 1 -- 172.21.15.146:0/979683322 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3806f500 con 0x7f3a380ae660 2024-04-25T15:22:34.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.385+0000 7f3a36d9d640 1 --2- 172.21.15.146:0/979683322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a380b4b50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3a380708b0 tx=0x7f3a3002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf645264a92c51b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.385+0000 7f3a3559a640 1 -- 172.21.15.146:0/979683322 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30037070 con 0x7f3a380ae660 2024-04-25T15:22:34.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.385+0000 7f3a3559a640 1 -- 172.21.15.146:0/979683322 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30032440 con 0x7f3a380ae660 2024-04-25T15:22:34.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.385+0000 7f3a3559a640 1 -- 172.21.15.146:0/979683322 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3002ae60 con 0x7f3a380ae660 2024-04-25T15:22:34.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 -- 172.21.15.146:0/979683322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 msgr2=0x7f3a380b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/979683322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a380b4b50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3a380708b0 tx=0x7f3a3002a110 comp rx=0 tx=0).stop 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 -- 172.21.15.146:0/979683322 shutdown_connections 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/979683322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a380b4b50 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/979683322 >> v2:172.21.15.177:3300/0 conn(0x7f3a380741f0 0x7f3a380b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 -- 172.21.15.146:0/979683322 >> 172.21.15.146:0/979683322 conn(0x7f3a380d1c10 msgr2=0x7f3a380733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 -- 172.21.15.146:0/979683322 shutdown_connections 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.386+0000 7f3a3ca92640 1 -- 172.21.15.146:0/979683322 wait complete. 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.388+0000 7f3a3ca92640 1 Processor -- start 2024-04-25T15:22:34.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.388+0000 7f3a3ca92640 1 -- start start 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3ca92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a3814c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3ca92640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3a3809ef10 0x7f3a38148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3ca92640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a381f4230 0x7f3a381f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3ca92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a380d0f10 con 0x7f3a380ae660 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3ca92640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3806fac0 con 0x7f3a381f4230 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3ca92640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3a381f7ab0 con 0x7f3a3809ef10 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a3814c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a3814c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50750/0 (socket says 172.21.15.146:50750) 2024-04-25T15:22:34.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a3659c640 1 -- 172.21.15.146:0/3559883304 learned_addr learned my addr 172.21.15.146:0/3559883304 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a35d9b640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3a3809ef10 0x7f3a38148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.389+0000 7f3a36d9d640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a381f4230 0x7f3a381f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3659c640 1 -- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3a3809ef10 msgr2=0x7f3a38148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3659c640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3a3809ef10 0x7f3a38148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3659c640 1 -- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a381f4230 msgr2=0x7f3a381f6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3659c640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a381f4230 0x7f3a381f6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3659c640 1 -- 172.21.15.146:0/3559883304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3814d090 con 0x7f3a380ae660 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a36d9d640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a381f4230 0x7f3a381f6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a35d9b640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3a3809ef10 0x7f3a38148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3659c640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a3814c210 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800c7f0 tx=0x7f3a2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28018070 con 0x7f3a380ae660 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a2800a640 con 0x7f3a380ae660 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2800aa30 con 0x7f3a380ae660 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3814cad0 con 0x7f3a380ae660 2024-04-25T15:22:34.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.390+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a381f3a30 con 0x7f3a380ae660 2024-04-25T15:22:34.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.391+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3806a0e0 con 0x7f3a380ae660 2024-04-25T15:22:34.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.392+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f3a2801b070 con 0x7f3a380ae660 2024-04-25T15:22:34.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.392+0000 7f3a26ffd640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3a180745d0 0x7f3a18076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.392+0000 7f3a35d9b640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3a180745d0 0x7f3a18076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.392+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f3a28094b80 con 0x7f3a380ae660 2024-04-25T15:22:34.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.393+0000 7f3a35d9b640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3a180745d0 0x7f3a18076ae0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c006fd0 tx=0x7f3a2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.397+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a2805f0d0 con 0x7f3a380ae660 2024-04-25T15:22:34.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.571+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3a38070c50 con 0x7f3a380ae660 2024-04-25T15:22:34.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.571+0000 7f3a26ffd640 1 -- 172.21.15.146:0/3559883304 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f3a280670c0 con 0x7f3a380ae660 2024-04-25T15:22:34.573 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3a180745d0 msgr2=0x7f3a18076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3a180745d0 0x7f3a18076ae0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c006fd0 tx=0x7f3a2c008040 comp rx=0 tx=0).stop 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 msgr2=0x7f3a3814c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a3814c210 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800c7f0 tx=0x7f3a2800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 shutdown_connections 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3a180745d0 0x7f3a18076ae0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a381f4230 0x7f3a381f6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3a3809ef10 0x7f3a38148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 --2- 172.21.15.146:0/3559883304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a380ae660 0x7f3a3814c210 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 >> 172.21.15.146:0/3559883304 conn(0x7f3a380d1c10 msgr2=0x7f3a3815a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:34.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 shutdown_connections 2024-04-25T15:22:34.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.573+0000 7f3a3ca92640 1 -- 172.21.15.146:0/3559883304 wait complete. 2024-04-25T15:22:34.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:34.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.698+0000 7f2cf4b7a640 1 Processor -- start 2024-04-25T15:22:34.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.698+0000 7f2cf4b7a640 1 -- start start 2024-04-25T15:22:34.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cf4b7a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cf00b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cf4b7a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2cf0163eb0 0x7f2cf01642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cf4b7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf00b8eb0 con 0x7f2cf0164800 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cf4b7a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf008d730 con 0x7f2cf0163eb0 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cf4b7a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf006f440 con 0x7f2cf006ee60 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ceed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2cf0164800 0x7f2cf0160830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32852/0 (socket says 172.21.15.146:32852) 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ceed9d640 1 -- 172.21.15.146:0/797293978 learned_addr learned my addr 172.21.15.146:0/797293978 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/797293978 >> v2:172.21.15.177:3300/0 conn(0x7f2cf0163eb0 0x7f2cf01642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cee59c640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cf00b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031721057 0 0) 0x7f2cf00b8eb0 con 0x7f2cf0164800 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd8003520 con 0x7f2cf0164800 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1559015829 0 0) 0x7f2cd8003520 con 0x7f2cf0164800 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 msgr2=0x7f2cf00b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:34.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cf00b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 >> v2:172.21.15.177:3300/0 conn(0x7f2cf0163eb0 msgr2=0x7f2cf01642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 --2- 172.21.15.146:0/797293978 >> v2:172.21.15.177:3300/0 conn(0x7f2cf0163eb0 0x7f2cf01642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf008ce00 con 0x7f2cf0164800 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f2ce8001670 con 0x7f2cf0164800 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/797293978 >> v2:172.21.15.177:3300/0 conn(0x7f2cf0163eb0 0x7f2cf01642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.700+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 >> v1:172.21.15.146:6789/0 conn(0x7f2cf0164800 legacy=0x7f2cf0160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ced59a640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd80039b0 0x7f2cd8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ced59a640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd8004800 0x7f2cd8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ced59a640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cd8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf008d730 con 0x7f2cd8004800 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf006f440 con 0x7f2cd80039b0 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2cd800b9c0 con 0x7f2cf006ee60 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd8004800 0x7f2cd8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cee59c640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd80039b0 0x7f2cd8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ceed9d640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cd8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cedd9b640 1 -- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 msgr2=0x7f2cd8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cd8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cedd9b640 1 -- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd80039b0 msgr2=0x7f2cd8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd80039b0 0x7f2cd8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2cedd9b640 1 -- 172.21.15.146:0/797293978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd8003520 con 0x7f2cd8004800 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.701+0000 7f2ceed9d640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cf006ee60 0x7f2cd8005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd8004800 0x7f2cd8004c10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009c90 tx=0x7f2cd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e136592be970d00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4036070 con 0x7f2cd8004800 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce4031400 con 0x7f2cd8004800 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2ced59a640 1 -- 172.21.15.146:0/797293978 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4009860 con 0x7f2cd8004800 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/797293978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd8004800 msgr2=0x7f2cd8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/797293978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd8004800 0x7f2cd8004c10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009c90 tx=0x7f2cd8005830 comp rx=0 tx=0).stop 2024-04-25T15:22:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.702+0000 7f2ceed9d640 1 -- 172.21.15.146:0/797293978 reap_dead start 2024-04-25T15:22:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.703+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/797293978 shutdown_connections 2024-04-25T15:22:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.703+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/797293978 >> 172.21.15.146:0/797293978 conn(0x7f2cf00b52e0 msgr2=0x7f2cf00b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.703+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/797293978 shutdown_connections 2024-04-25T15:22:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.703+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/797293978 wait complete. 2024-04-25T15:22:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.704+0000 7f2cf4b7a640 1 Processor -- start 2024-04-25T15:22:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.705+0000 7f2cf4b7a640 1 -- start start 2024-04-25T15:22:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cf4b7a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cd8004800 0x7f2cf009d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cf4b7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 0x7f2cf00b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cf4b7a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cf014c0c0 0x7f2cf014c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cf4b7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf008dbc0 con 0x7f2cf009f040 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cf4b7a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf008d3c0 con 0x7f2cf014c0c0 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cf4b7a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00d2b70 con 0x7f2cd8004800 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 0x7f2cf00b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 0x7f2cf00b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50762/0 (socket says 172.21.15.146:50762) 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 -- 172.21.15.146:0/3429999123 learned_addr learned my addr 172.21.15.146:0/3429999123 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2ceed9d640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cf014c0c0 0x7f2cf014c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cee59c640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cd8004800 0x7f2cf009d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 -- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cd8004800 msgr2=0x7f2cf009d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cd8004800 0x7f2cf009d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 -- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cf014c0c0 msgr2=0x7f2cf014c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cf014c0c0 0x7f2cf014c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 -- 172.21.15.146:0/3429999123 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf00b4500 con 0x7f2cf009f040 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cedd9b640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 0x7f2cf00b3fc0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009c90 tx=0x7f2ce4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4040020 con 0x7f2cf009f040 2024-04-25T15:22:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.706+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce403b680 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.707+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf015deb0 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.707+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4036040 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.707+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf014b300 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.708+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf014ad90 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.708+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f2ce4043020 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.709+0000 7f2cdeffd640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2cd00745d0 0x7f2cd0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.709+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f2ce40b5d80 con 0x7f2cf009f040 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.709+0000 7f2cee59c640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2cd00745d0 0x7f2cd0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.709+0000 7f2cee59c640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2cd00745d0 0x7f2cd0076ae0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007440 tx=0x7f2ce000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.713+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ce404a030 con 0x7f2cf009f040 2024-04-25T15:22:34.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.900+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2cf00721f0 con 0x7f2cf009f040 2024-04-25T15:22:34.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.901+0000 7f2cdeffd640 1 -- 172.21.15.146:0/3429999123 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f2cf00721f0 con 0x7f2cf009f040 2024-04-25T15:22:34.902 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:22:34.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.902+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2cd00745d0 msgr2=0x7f2cd0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.902+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2cd00745d0 0x7f2cd0076ae0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007440 tx=0x7f2ce000b040 comp rx=0 tx=0).stop 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.902+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 msgr2=0x7f2cf00b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.902+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 0x7f2cf00b3fc0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009c90 tx=0x7f2ce4004730 comp rx=0 tx=0).stop 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.902+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 shutdown_connections 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.902+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2cd00745d0 0x7f2cd0076ae0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.903+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cf014c0c0 0x7f2cf014c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.903+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf009f040 0x7f2cf00b3fc0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.903+0000 7f2cf4b7a640 1 --2- 172.21.15.146:0/3429999123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2cd8004800 0x7f2cf009d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.903+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 >> 172.21.15.146:0/3429999123 conn(0x7f2cf00b52e0 msgr2=0x7f2cf00b8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.903+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 shutdown_connections 2024-04-25T15:22:34.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:34.903+0000 7f2cf4b7a640 1 -- 172.21.15.146:0/3429999123 wait complete. 2024-04-25T15:22:34.915 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:22:34.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:34.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:34.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[57653]: fuse finished with error 0 and tester_r 0 2024-04-25T15:22:34.939 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:22:35.035 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:22:35.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:35.035 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:22:35.050 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:35.050 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:35.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:35.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.197+0000 7f1c1317a640 1 Processor -- start 2024-04-25T15:22:35.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.197+0000 7f1c1317a640 1 -- start start 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c1317a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c0c06ee60 0x7f1c0c0b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c1317a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1c0c1649d0 0x7f1c0c0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c1317a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c0b8eb0 con 0x7f1c0c163eb0 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c1317a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1c0c161840 con 0x7f1c0c06ee60 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c1317a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f1c0c164de0 con 0x7f1c0c1649d0 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c0c06ee60 0x7f1c0c0b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c0c06ee60 0x7f1c0c0b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50764/0 (socket says 172.21.15.146:50764) 2024-04-25T15:22:35.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 -- 172.21.15.146:0/3111062238 learned_addr learned my addr 172.21.15.146:0/3111062238 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:35.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c11717640 1 --2- 172.21.15.146:0/3111062238 >> v2:172.21.15.177:3300/0 conn(0x7f1c0c1649d0 0x7f1c0c0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 -- 172.21.15.146:0/3111062238 >> v2:172.21.15.177:3300/0 conn(0x7f1c0c1649d0 msgr2=0x7f1c0c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:35.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 --2- 172.21.15.146:0/3111062238 >> v2:172.21.15.177:3300/0 conn(0x7f1c0c1649d0 0x7f1c0c0d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 -- 172.21.15.146:0/3111062238 >> v1:172.21.15.177:6789/0 conn(0x7f1c0c163eb0 legacy=0x7f1c0c1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:35.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.198+0000 7f1c10f16640 1 -- 172.21.15.146:0/3111062238 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c08ce00 con 0x7f1c0c06ee60 2024-04-25T15:22:35.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.199+0000 7f1c10f16640 1 --2- 172.21.15.146:0/3111062238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c0b2ba0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0b5290 tx=0x7f1bfc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f16ab2ad658893 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.199+0000 7f1c0b7fe640 1 -- 172.21.15.146:0/3111062238 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc004030 con 0x7f1c0c06ee60 2024-04-25T15:22:35.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.199+0000 7f1c0b7fe640 1 -- 172.21.15.146:0/3111062238 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bfc0041d0 con 0x7f1c0c06ee60 2024-04-25T15:22:35.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.199+0000 7f1c0b7fe640 1 -- 172.21.15.146:0/3111062238 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc003610 con 0x7f1c0c06ee60 2024-04-25T15:22:35.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 -- 172.21.15.146:0/3111062238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 msgr2=0x7f1c0c0b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 --2- 172.21.15.146:0/3111062238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c0b2ba0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0b5290 tx=0x7f1bfc02a230 comp rx=0 tx=0).stop 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 -- 172.21.15.146:0/3111062238 shutdown_connections 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 --2- 172.21.15.146:0/3111062238 >> v2:172.21.15.177:3300/0 conn(0x7f1c0c1649d0 0x7f1c0c0d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 --2- 172.21.15.146:0/3111062238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c0b2ba0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 -- 172.21.15.146:0/3111062238 >> 172.21.15.146:0/3111062238 conn(0x7f1c0c0b52e0 msgr2=0x7f1c0c0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 -- 172.21.15.146:0/3111062238 shutdown_connections 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.200+0000 7f1c1317a640 1 -- 172.21.15.146:0/3111062238 wait complete. 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.202+0000 7f1c1317a640 1 Processor -- start 2024-04-25T15:22:35.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.202+0000 7f1c1317a640 1 -- start start 2024-04-25T15:22:35.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c1317a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c1317a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c0c163eb0 0x7f1c0c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c1317a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c0c1649d0 0x7f1c0c0b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c1317a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c06f440 con 0x7f1c0c06ee60 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c1317a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c08d730 con 0x7f1c0c1649d0 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c1317a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c08d3c0 con 0x7f1c0c163eb0 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c10f16640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c10f16640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50772/0 (socket says 172.21.15.146:50772) 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.203+0000 7f1c10f16640 1 -- 172.21.15.146:0/1386259282 learned_addr learned my addr 172.21.15.146:0/1386259282 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c11717640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c0c1649d0 0x7f1c0c0b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c0bfff640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c0c163eb0 0x7f1c0c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c10f16640 1 -- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c0c163eb0 msgr2=0x7f1c0c0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c10f16640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c0c163eb0 0x7f1c0c0b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c10f16640 1 -- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c0c1649d0 msgr2=0x7f1c0c0b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c10f16640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c0c1649d0 0x7f1c0c0b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c10f16640 1 -- 172.21.15.146:0/1386259282 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c0b0950 con 0x7f1c0c06ee60 2024-04-25T15:22:35.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c10f16640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c151e70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc041020 con 0x7f1c0c06ee60 2024-04-25T15:22:35.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bfc003c50 con 0x7f1c0c06ee60 2024-04-25T15:22:35.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c0b0c30 con 0x7f1c0c06ee60 2024-04-25T15:22:35.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc004030 con 0x7f1c0c06ee60 2024-04-25T15:22:35.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.204+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c0b11f0 con 0x7f1c0c06ee60 2024-04-25T15:22:35.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.205+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0c06a0e0 con 0x7f1c0c06ee60 2024-04-25T15:22:35.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.206+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f1bfc044020 con 0x7f1c0c06ee60 2024-04-25T15:22:35.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.206+0000 7f1c097fa640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1bf80745d0 0x7f1bf8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.207+0000 7f1c0bfff640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1bf80745d0 0x7f1bf8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.207+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f1bfc0b60e0 con 0x7f1c0c06ee60 2024-04-25T15:22:35.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.207+0000 7f1c0bfff640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1bf80745d0 0x7f1bf8076ae0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c1eb1b0 tx=0x7f1c00009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.211+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1bfc04b030 con 0x7f1c0c06ee60 2024-04-25T15:22:35.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.392+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c0c0b8640 con 0x7f1c0c06ee60 2024-04-25T15:22:35.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.393+0000 7f1c097fa640 1 -- 172.21.15.146:0/1386259282 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+17029 (secure 0 0 0) 0x7f1bfc07ff50 con 0x7f1c0c06ee60 2024-04-25T15:22:35.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:35.395 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":56,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:28.267703+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1bf80745d0 msgr2=0x7f1bf8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1bf80745d0 0x7f1bf8076ae0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c1eb1b0 tx=0x7f1c00009060 comp rx=0 tx=0).stop 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 msgr2=0x7f1c0c151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c151e70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 shutdown_connections 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1bf80745d0 0x7f1bf8076ae0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c0c1649d0 0x7f1c0c0b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c0c163eb0 0x7f1c0c0b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 --2- 172.21.15.146:0/1386259282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ee60 0x7f1c0c151e70 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 >> 172.21.15.146:0/1386259282 conn(0x7f1c0c0b52e0 msgr2=0x7f1c0c0cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:35.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 shutdown_connections 2024-04-25T15:22:35.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.395+0000 7f1c1317a640 1 -- 172.21.15.146:0/1386259282 wait complete. 2024-04-25T15:22:35.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:35.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:35.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.518+0000 7eff0211c640 1 Processor -- start 2024-04-25T15:22:35.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.518+0000 7eff0211c640 1 -- start start 2024-04-25T15:22:35.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7eff0211c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efefc0d3300 0x7efefc0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7eff0211c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efefc0d2260 con 0x7efefc071d30 2024-04-25T15:22:35.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7eff0211c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efefc0d2540 con 0x7efefc0b2cd0 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7eff0211c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7efefc0d2820 con 0x7efefc0d3300 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefaffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efefc071d30 0x7efefc0cde40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32866/0 (socket says 172.21.15.146:32866) 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefaffd640 1 -- 172.21.15.146:0/3048107839 learned_addr learned my addr 172.21.15.146:0/3048107839 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefbfff640 1 --2- 172.21.15.146:0/3048107839 >> v2:172.21.15.177:3300/0 conn(0x7efefc0d3300 0x7efefc0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2422706513 0 0) 0x7efefc0d2260 con 0x7efefc071d30 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efee4003520 con 0x7efefc071d30 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 180999503 0 0) 0x7efee4003520 con 0x7efefc071d30 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 >> v2:172.21.15.177:3300/0 conn(0x7efefc0d3300 msgr2=0x7efefc0d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 --2- 172.21.15.146:0/3048107839 >> v2:172.21.15.177:3300/0 conn(0x7efefc0d3300 0x7efefc0d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 >> v1:172.21.15.177:6789/0 conn(0x7efefc0b2cd0 legacy=0x7efefc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc0d2b00 con 0x7efefc071d30 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.520+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7efef0001670 con 0x7efefc071d30 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 >> v1:172.21.15.146:6789/0 conn(0x7efefc071d30 legacy=0x7efefc0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee40039b0 0x7efee4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efee4004800 0x7efee4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee4005250 0x7efee400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efefc0d2820 con 0x7efee4004800 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efee400c1e0 con 0x7efee4005250 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efee400c3e0 con 0x7efee40039b0 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefaffd640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efee4004800 0x7efee4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefb7fe640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee40039b0 0x7efee4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefbfff640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee4005250 0x7efee400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefaffd640 1 -- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee40039b0 msgr2=0x7efee4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefaffd640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee40039b0 0x7efee4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefaffd640 1 -- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee4005250 msgr2=0x7efee400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefaffd640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee4005250 0x7efee400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefaffd640 1 -- 172.21.15.146:0/3048107839 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee4003520 con 0x7efee4004800 2024-04-25T15:22:35.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefbfff640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee4005250 0x7efee400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefbfff640 1 -- 172.21.15.146:0/3048107839 reap_dead start 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.521+0000 7efefb7fe640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee40039b0 0x7efee4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7efefaffd640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efee4004800 0x7efee4004c10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efef000c210 tx=0x7efee4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4bf2882913ccf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0007d30 con 0x7efee4004800 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef00058d0 con 0x7efee4004800 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7efefa7fc640 1 -- 172.21.15.146:0/3048107839 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0005d20 con 0x7efee4004800 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7eff0211c640 1 -- 172.21.15.146:0/3048107839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efee4004800 msgr2=0x7efee4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7eff0211c640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efee4004800 0x7efee4004c10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efef000c210 tx=0x7efee4005830 comp rx=0 tx=0).stop 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7eff0211c640 1 -- 172.21.15.146:0/3048107839 shutdown_connections 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7eff0211c640 1 --2- 172.21.15.146:0/3048107839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efee4004800 0x7efee4004c10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.522+0000 7eff0211c640 1 -- 172.21.15.146:0/3048107839 >> 172.21.15.146:0/3048107839 conn(0x7efefc0b52e0 msgr2=0x7efefc0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.523+0000 7eff0211c640 1 -- 172.21.15.146:0/3048107839 shutdown_connections 2024-04-25T15:22:35.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.523+0000 7eff0211c640 1 -- 172.21.15.146:0/3048107839 wait complete. 2024-04-25T15:22:35.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.524+0000 7eff0211c640 1 Processor -- start 2024-04-25T15:22:35.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.524+0000 7eff0211c640 1 -- start start 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee4004800 0x7efefc163020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc152e10 0x7efefc0b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 0x7efefc0cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efefc08dbf0 con 0x7efefc0b1950 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efefc0cfb70 con 0x7efefc152e10 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7efefc08dd70 con 0x7efee4004800 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 0x7efefc0cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 0x7efefc0cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50798/0 (socket says 172.21.15.146:50798) 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 -- 172.21.15.146:0/3089631191 learned_addr learned my addr 172.21.15.146:0/3089631191 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefb7fe640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee4004800 0x7efefc163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefaffd640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc152e10 0x7efefc0b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 -- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee4004800 msgr2=0x7efefc163020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee4004800 0x7efefc163020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 -- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc152e10 msgr2=0x7efefc0b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc152e10 0x7efefc0b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 -- 172.21.15.146:0/3089631191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc0b1d60 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efefbfff640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 0x7efefc0cbb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7efefc0710e0 tx=0x7efef4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4010da0 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef4003990 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc160ae0 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.526+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4003d80 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.527+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc1f9900 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.528+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efefc06a0e0 con 0x7efefc0b1950 2024-04-25T15:22:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.528+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7efef401e020 con 0x7efefc0b1950 2024-04-25T15:22:35.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.528+0000 7efee3fff640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efee8074600 0x7efee8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.529+0000 7efefb7fe640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efee8074600 0x7efee8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.529+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7efef4095d10 con 0x7efefc0b1950 2024-04-25T15:22:35.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.529+0000 7efefb7fe640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efee8074600 0x7efee8076b10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efeec000c00 tx=0x7efeec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.533+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efefc0b1d60 con 0x7efefc0b1950 2024-04-25T15:22:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.711+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efefc0b31e0 con 0x7efefc0b1950 2024-04-25T15:22:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.711+0000 7efee3fff640 1 -- 172.21.15.146:0/3089631191 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7efef405fb00 con 0x7efefc0b1950 2024-04-25T15:22:35.713 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efee8074600 msgr2=0x7efee8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efee8074600 0x7efee8076b10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efeec000c00 tx=0x7efeec009040 comp rx=0 tx=0).stop 2024-04-25T15:22:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 msgr2=0x7efefc0cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 0x7efefc0cbb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7efefc0710e0 tx=0x7efef4010590 comp rx=0 tx=0).stop 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 shutdown_connections 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7efee8074600 0x7efee8076b10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc0b1950 0x7efefc0cbb80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc152e10 0x7efefc0b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 --2- 172.21.15.146:0/3089631191 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efee4004800 0x7efefc163020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 >> 172.21.15.146:0/3089631191 conn(0x7efefc0b52e0 msgr2=0x7efefc0b87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 shutdown_connections 2024-04-25T15:22:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.713+0000 7eff0211c640 1 -- 172.21.15.146:0/3089631191 wait complete. 2024-04-25T15:22:35.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:35.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.836+0000 7f3e0c014640 1 Processor -- start 2024-04-25T15:22:35.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.836+0000 7f3e0c014640 1 -- start start 2024-04-25T15:22:35.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0c014640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e040b2cd0 0x7f3e040d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0c014640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e04071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0c014640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e04071f70 con 0x7f3e04071b60 2024-04-25T15:22:35.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0c014640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3e040d2260 con 0x7f3e040b2cd0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0c014640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3e040d2540 con 0x7f3e040d3300 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e09db0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e040b2cd0 0x7f3e040d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0a5b1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3e04071b60 0x7f3e0408cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32870/0 (socket says 172.21.15.146:32870) 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e0a5b1640 1 -- 172.21.15.146:0/2940807971 learned_addr learned my addr 172.21.15.146:0/2940807971 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e09db0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e040b2cd0 0x7f3e040d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50806/0 (socket says 172.21.15.146:50806) 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e095af640 1 --2- 172.21.15.146:0/2940807971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e04071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e09db0640 1 -- 172.21.15.146:0/2940807971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 msgr2=0x7f3e04071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e09db0640 1 --2- 172.21.15.146:0/2940807971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e04071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e09db0640 1 -- 172.21.15.146:0/2940807971 >> v1:172.21.15.146:6789/0 conn(0x7f3e04071b60 legacy=0x7f3e0408cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e09db0640 1 -- 172.21.15.146:0/2940807971 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e040d2820 con 0x7f3e040b2cd0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.838+0000 7f3e08dae640 1 -- 172.21.15.146:0/2940807971 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486777744 0 0) 0x7f3dfc001490 con 0x7f3e04071b60 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.839+0000 7f3e09db0640 1 --2- 172.21.15.146:0/2940807971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040d1c80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009790 tx=0x7f3e0002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919722546c438acf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.839+0000 7f3e08dae640 1 -- 172.21.15.146:0/2940807971 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00037070 con 0x7f3e040b2cd0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.839+0000 7f3e08dae640 1 -- 172.21.15.146:0/2940807971 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e0002ae50 con 0x7f3e040b2cd0 2024-04-25T15:22:35.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.839+0000 7f3e08dae640 1 -- 172.21.15.146:0/2940807971 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00032660 con 0x7f3e040b2cd0 2024-04-25T15:22:35.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 -- 172.21.15.146:0/2940807971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 msgr2=0x7f3e040d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2940807971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040d1c80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009790 tx=0x7f3e0002a100 comp rx=0 tx=0).stop 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 -- 172.21.15.146:0/2940807971 shutdown_connections 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2940807971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e04071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2940807971 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040d1c80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 -- 172.21.15.146:0/2940807971 >> 172.21.15.146:0/2940807971 conn(0x7f3e040b52e0 msgr2=0x7f3e040b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 -- 172.21.15.146:0/2940807971 shutdown_connections 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.840+0000 7f3e0c014640 1 -- 172.21.15.146:0/2940807971 wait complete. 2024-04-25T15:22:35.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.841+0000 7f3e0c014640 1 Processor -- start 2024-04-25T15:22:35.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.841+0000 7f3e0c014640 1 -- start start 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04071b60 0x7f3e040cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e040cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3e040cfb70 con 0x7f3e040b2cd0 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e040d2de0 con 0x7f3e04071b60 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0408dbf0 con 0x7f3e040d3300 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50812/0 (socket says 172.21.15.146:50812) 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 -- 172.21.15.146:0/2521920977 learned_addr learned my addr 172.21.15.146:0/2521920977 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:35.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e09db0640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04071b60 0x7f3e040cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0a5b1640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e040cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 -- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 msgr2=0x7f3e040cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e040cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 -- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04071b60 msgr2=0x7f3e040cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04071b60 0x7f3e040cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 -- 172.21.15.146:0/2521920977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e040c7550 con 0x7f3e040b2cd0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0a5b1640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e040cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e095af640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040c7d10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3df400d620 tx=0x7f3df400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df400b620 con 0x7f3e040b2cd0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df4019070 con 0x7f3e040b2cd0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df400ba00 con 0x7f3e040b2cd0 2024-04-25T15:22:35.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.843+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e040c7830 con 0x7f3e040b2cd0 2024-04-25T15:22:35.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.844+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e040cc340 con 0x7f3e040b2cd0 2024-04-25T15:22:35.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.844+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e040d1860 con 0x7f3e040b2cd0 2024-04-25T15:22:35.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.845+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f3df400bba0 con 0x7f3e040b2cd0 2024-04-25T15:22:35.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.845+0000 7f3dfa7fc640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3de8074600 0x7f3de8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:35.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.846+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f3df4095880 con 0x7f3e040b2cd0 2024-04-25T15:22:35.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.846+0000 7f3e09db0640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3de8074600 0x7f3de8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:35.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.846+0000 7f3e09db0640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3de8074600 0x7f3de8076b10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3e00034000 tx=0x7f3e0002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:35.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:35.850+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e040d1860 con 0x7f3e040b2cd0 2024-04-25T15:22:36.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.027+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3e0406a0e0 con 0x7f3e040b2cd0 2024-04-25T15:22:36.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.028+0000 7f3dfa7fc640 1 -- 172.21.15.146:0/2521920977 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f3df405fdd0 con 0x7f3e040b2cd0 2024-04-25T15:22:36.029 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3de8074600 msgr2=0x7f3de8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3de8074600 0x7f3de8076b10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3e00034000 tx=0x7f3e0002e000 comp rx=0 tx=0).stop 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 msgr2=0x7f3e040c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040c7d10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3df400d620 tx=0x7f3df400dae0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 shutdown_connections 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3de8074600 0x7f3de8076b10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e040d3300 0x7f3e040cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e040b2cd0 0x7f3e040c7d10 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 --2- 172.21.15.146:0/2521920977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04071b60 0x7f3e040cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 >> 172.21.15.146:0/2521920977 conn(0x7f3e040b52e0 msgr2=0x7f3e0406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 shutdown_connections 2024-04-25T15:22:36.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.030+0000 7f3e0c014640 1 -- 172.21.15.146:0/2521920977 wait complete. 2024-04-25T15:22:36.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:36.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.151+0000 7ff9ef97a640 1 Processor -- start 2024-04-25T15:22:36.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.152+0000 7ff9ef97a640 1 -- start start 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ef97a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9e80b2310 0x7ff9e80d24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ef97a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff9e80d2a10 0x7ff9e80d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ef97a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9e80d2e20 con 0x7ff9e80b7430 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ef97a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff9e80b3820 con 0x7ff9e80b2310 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ef97a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff9e80b2fe0 con 0x7ff9e80d2a10 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9e80b2310 0x7ff9e80d24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9e80b2310 0x7ff9e80d24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50828/0 (socket says 172.21.15.146:50828) 2024-04-25T15:22:36.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 -- 172.21.15.146:0/260961760 learned_addr learned my addr 172.21.15.146:0/260961760 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:36.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9edf17640 1 --2- 172.21.15.146:0/260961760 >> v2:172.21.15.177:3300/0 conn(0x7ff9e80d2a10 0x7ff9e80d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 -- 172.21.15.146:0/260961760 >> v2:172.21.15.177:3300/0 conn(0x7ff9e80d2a10 msgr2=0x7ff9e80d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:36.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 --2- 172.21.15.146:0/260961760 >> v2:172.21.15.177:3300/0 conn(0x7ff9e80d2a10 0x7ff9e80d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 -- 172.21.15.146:0/260961760 >> v1:172.21.15.177:6789/0 conn(0x7ff9e80b7430 legacy=0x7ff9e806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.153+0000 7ff9ecf15640 1 -- 172.21.15.146:0/260961760 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e80d1ff0 con 0x7ff9e80b2310 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.154+0000 7ff9ecf15640 1 --2- 172.21.15.146:0/260961760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80b2310 0x7ff9e80d24d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80d1d70 tx=0x7ff9e402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ae0ad853dc43f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.154+0000 7ff9dbfff640 1 -- 172.21.15.146:0/260961760 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4037070 con 0x7ff9e80b2310 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.154+0000 7ff9dbfff640 1 -- 172.21.15.146:0/260961760 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e4032440 con 0x7ff9e80b2310 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.154+0000 7ff9dbfff640 1 -- 172.21.15.146:0/260961760 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4032810 con 0x7ff9e80b2310 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 -- 172.21.15.146:0/260961760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80b2310 msgr2=0x7ff9e80d24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/260961760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80b2310 0x7ff9e80d24d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80d1d70 tx=0x7ff9e402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 -- 172.21.15.146:0/260961760 shutdown_connections 2024-04-25T15:22:36.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/260961760 >> v2:172.21.15.177:3300/0 conn(0x7ff9e80d2a10 0x7ff9e80d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/260961760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80b2310 0x7ff9e80d24d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 -- 172.21.15.146:0/260961760 >> 172.21.15.146:0/260961760 conn(0x7ff9e809b2b0 msgr2=0x7ff9e80b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:36.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 -- 172.21.15.146:0/260961760 shutdown_connections 2024-04-25T15:22:36.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.155+0000 7ff9ef97a640 1 -- 172.21.15.146:0/260961760 wait complete. 2024-04-25T15:22:36.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.157+0000 7ff9ef97a640 1 Processor -- start 2024-04-25T15:22:36.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.157+0000 7ff9ef97a640 1 -- start start 2024-04-25T15:22:36.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9ef97a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e80b2310 0x7ff9e8155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9ef97a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9e80b7430 0x7ff9e81574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9ef97a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 0x7ff9e8154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9ef97a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e80b2720 con 0x7ff9e80d2a10 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9ef97a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e80b3540 con 0x7ff9e80b2310 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9ef97a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff9e806cc60 con 0x7ff9e80b7430 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.158+0000 7ff9edf17640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 0x7ff9e8154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 0x7ff9e8154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50842/0 (socket says 172.21.15.146:50842) 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 -- 172.21.15.146:0/4176716185 learned_addr learned my addr 172.21.15.146:0/4176716185 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9ecf15640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9e80b7430 0x7ff9e81574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9ed716640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e80b2310 0x7ff9e8155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 -- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9e80b7430 msgr2=0x7ff9e81574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9e80b7430 0x7ff9e81574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 -- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e80b2310 msgr2=0x7ff9e8155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e80b2310 0x7ff9e8155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 -- 172.21.15.146:0/4176716185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e8157a20 con 0x7ff9e80d2a10 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9ecf15640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9e80b7430 0x7ff9e81574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9ed716640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e80b2310 0x7ff9e8155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9edf17640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 0x7ff9e8154480 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000efc0 tx=0x7ff9e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0018020 con 0x7ff9e80d2a10 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e0014480 con 0x7ff9e80d2a10 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0014870 con 0x7ff9e80d2a10 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e8154f50 con 0x7ff9e80d2a10 2024-04-25T15:22:36.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.159+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e8155210 con 0x7ff9e80d2a10 2024-04-25T15:22:36.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.161+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e8157ca0 con 0x7ff9e80d2a10 2024-04-25T15:22:36.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.161+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7ff9e0005d10 con 0x7ff9e80d2a10 2024-04-25T15:22:36.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.162+0000 7ff9d9ffb640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9cc0745d0 0x7ff9cc076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.162+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7ff9e00963d0 con 0x7ff9e80d2a10 2024-04-25T15:22:36.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.162+0000 7ff9ed716640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9cc0745d0 0x7ff9cc076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.162+0000 7ff9ed716640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9cc0745d0 0x7ff9cc076ae0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc001660 tx=0x7ff9dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.166+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9e0022030 con 0x7ff9e80d2a10 2024-04-25T15:22:36.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.345+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff9e806a0e0 con 0x7ff9e80d2a10 2024-04-25T15:22:36.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.346+0000 7ff9d9ffb640 1 -- 172.21.15.146:0/4176716185 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7ff9e0010040 con 0x7ff9e80d2a10 2024-04-25T15:22:36.348 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9cc0745d0 msgr2=0x7ff9cc076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9cc0745d0 0x7ff9cc076ae0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc001660 tx=0x7ff9dc009040 comp rx=0 tx=0).stop 2024-04-25T15:22:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 msgr2=0x7ff9e8154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 0x7ff9e8154480 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000efc0 tx=0x7ff9e000c490 comp rx=0 tx=0).stop 2024-04-25T15:22:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 shutdown_connections 2024-04-25T15:22:36.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff9cc0745d0 0x7ff9cc076ae0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9e80d2a10 0x7ff9e8154480 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9e80b7430 0x7ff9e81574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 --2- 172.21.15.146:0/4176716185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9e80b2310 0x7ff9e8155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 >> 172.21.15.146:0/4176716185 conn(0x7ff9e809b2b0 msgr2=0x7ff9e80cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:36.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 shutdown_connections 2024-04-25T15:22:36.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.348+0000 7ff9ef97a640 1 -- 172.21.15.146:0/4176716185 wait complete. 2024-04-25T15:22:36.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:36.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.467+0000 7ff0f36f9640 1 Processor -- start 2024-04-25T15:22:36.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.467+0000 7ff0f36f9640 1 -- start start 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f36f9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0ec0cfbe0 0x7ff0ec0cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f36f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff0ec0d19d0 0x7ff0ec0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f36f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f36f9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ec0b5020 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f36f9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ec0b4a60 con 0x7ff0ec0d19d0 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f36f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ec0b4d40 con 0x7ff0ec0b3920 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0ec0cfbe0 0x7ff0ec0cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0ec0cfbe0 0x7ff0ec0cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50850/0 (socket says 172.21.15.146:50850) 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 -- 172.21.15.146:0/2747724091 learned_addr learned my addr 172.21.15.146:0/2747724091 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f0c94640 1 --2- 172.21.15.146:0/2747724091 >> v2:172.21.15.177:3300/0 conn(0x7ff0ec0d19d0 0x7ff0ec0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1c96640 1 --2- 172.21.15.146:0/2747724091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 -- 172.21.15.146:0/2747724091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 msgr2=0x7ff0ec0b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 --2- 172.21.15.146:0/2747724091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 -- 172.21.15.146:0/2747724091 >> v2:172.21.15.177:3300/0 conn(0x7ff0ec0d19d0 msgr2=0x7ff0ec0d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 --2- 172.21.15.146:0/2747724091 >> v2:172.21.15.177:3300/0 conn(0x7ff0ec0d19d0 0x7ff0ec0d1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.469+0000 7ff0f1495640 1 -- 172.21.15.146:0/2747724091 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ec0cead0 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.470+0000 7ff0f1495640 1 --2- 172.21.15.146:0/2747724091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec0cf260 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0b8510 tx=0x7ff0e802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25377c1426b668b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.470+0000 7ff0e3fff640 1 -- 172.21.15.146:0/2747724091 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80325f0 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.470+0000 7ff0e3fff640 1 -- 172.21.15.146:0/2747724091 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8032790 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.470+0000 7ff0e3fff640 1 -- 172.21.15.146:0/2747724091 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8032b60 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 -- 172.21.15.146:0/2747724091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 msgr2=0x7ff0ec0cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/2747724091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec0cf260 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0b8510 tx=0x7ff0e802a780 comp rx=0 tx=0).stop 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 -- 172.21.15.146:0/2747724091 shutdown_connections 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/2747724091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/2747724091 >> v2:172.21.15.177:3300/0 conn(0x7ff0ec0d19d0 0x7ff0ec0d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/2747724091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec0cf260 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 -- 172.21.15.146:0/2747724091 >> 172.21.15.146:0/2747724091 conn(0x7ff0ec0b73b0 msgr2=0x7ff0ec0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 -- 172.21.15.146:0/2747724091 shutdown_connections 2024-04-25T15:22:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.471+0000 7ff0f36f9640 1 -- 172.21.15.146:0/2747724091 wait complete. 2024-04-25T15:22:36.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.473+0000 7ff0f36f9640 1 Processor -- start 2024-04-25T15:22:36.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.473+0000 7ff0f36f9640 1 -- start start 2024-04-25T15:22:36.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f36f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f36f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec15df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f36f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0ec0d19d0 0x7ff0ec0af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f36f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ec0cf7a0 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f36f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ec0d2320 con 0x7ff0ec0d19d0 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f36f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ec0aff80 con 0x7ff0ec0b3920 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec15df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50860/0 (socket says 172.21.15.146:50860) 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 -- 172.21.15.146:0/3804162613 learned_addr learned my addr 172.21.15.146:0/3804162613 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f1c96640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0ec0d19d0 0x7ff0ec0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f1495640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 -- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 msgr2=0x7ff0ec0c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 -- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0ec0d19d0 msgr2=0x7ff0ec0af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0ec0d19d0 0x7ff0ec0af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 -- 172.21.15.146:0/3804162613 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ec15e450 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.474+0000 7ff0f0c94640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec15df10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00c7f0 tx=0x7ff0dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.475+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc004230 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.475+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0043d0 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.475+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc007860 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.475+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ec0c6440 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.475+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ec0c6a00 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.476+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ec0cc790 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.476+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7ff0dc007c10 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.477+0000 7ff0e1ffb640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff0d00745d0 0x7ff0d0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.477+0000 7ff0f1495640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff0d00745d0 0x7ff0d0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.477+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7ff0dc0957d0 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.477+0000 7ff0f1495640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff0d00745d0 0x7ff0d0076ae0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8033000 tx=0x7ff0e802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.481+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0ec0cc790 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.671+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff0ec06a0e0 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.672+0000 7ff0e1ffb640 1 -- 172.21.15.146:0/3804162613 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7ff0dc015020 con 0x7ff0ec0cfbe0 2024-04-25T15:22:36.674 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff0d00745d0 msgr2=0x7ff0d0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff0d00745d0 0x7ff0d0076ae0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8033000 tx=0x7ff0e802e000 comp rx=0 tx=0).stop 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 msgr2=0x7ff0ec15df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec15df10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00c7f0 tx=0x7ff0dc00ccb0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 shutdown_connections 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff0d00745d0 0x7ff0d0076ae0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0ec0d19d0 0x7ff0ec0af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ec0cfbe0 0x7ff0ec15df10 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 --2- 172.21.15.146:0/3804162613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0ec0b3920 0x7ff0ec0c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 >> 172.21.15.146:0/3804162613 conn(0x7ff0ec0b73b0 msgr2=0x7ff0ec0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 shutdown_connections 2024-04-25T15:22:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.674+0000 7ff0f36f9640 1 -- 172.21.15.146:0/3804162613 wait complete. 2024-04-25T15:22:36.687 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:36.687 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:36.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:36.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.812+0000 7f9a91384640 1 Processor -- start 2024-04-25T15:22:36.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.812+0000 7f9a91384640 1 -- start start 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.813+0000 7f9a91384640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c0b6610 0x7f9a8c0d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.813+0000 7f9a91384640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a8c0b5950 0x7f9a8c0b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.813+0000 7f9a91384640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8c0d2230 con 0x7f9a8c15ffa0 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.813+0000 7f9a91384640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8c0b2f50 con 0x7f9a8c0b5950 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.813+0000 7f9a91384640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c0b2ac0 con 0x7f9a8c0b6610 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.813+0000 7f9a8b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a8c0b5950 0x7f9a8c0b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a8c0b5950 0x7f9a8c0b7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50866/0 (socket says 172.21.15.146:50866) 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8b7fe640 1 -- 172.21.15.146:0/745455941 learned_addr learned my addr 172.21.15.146:0/745455941 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8affd640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c0b6610 0x7f9a8c0d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8b7fe640 1 -- 172.21.15.146:0/745455941 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c0b6610 msgr2=0x7f9a8c0d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8b7fe640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c0b6610 0x7f9a8c0d2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8b7fe640 1 -- 172.21.15.146:0/745455941 >> v1:172.21.15.177:6789/0 conn(0x7f9a8c15ffa0 legacy=0x7f9a8c0b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8b7fe640 1 -- 172.21.15.146:0/745455941 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c0cfa80 con 0x7f9a8c0b5950 2024-04-25T15:22:36.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.814+0000 7f9a8affd640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c0b6610 0x7f9a8c0d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a8b7fe640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b5950 0x7f9a8c0b7cc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009e50 tx=0x7f9a8402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b461364962e413f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a8a7fc640 1 -- 172.21.15.146:0/745455941 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84036070 con 0x7f9a8c0b5950 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a8a7fc640 1 -- 172.21.15.146:0/745455941 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a84031600 con 0x7f9a8c0b5950 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a8a7fc640 1 -- 172.21.15.146:0/745455941 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a840319d0 con 0x7f9a8c0b5950 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 -- 172.21.15.146:0/745455941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b5950 msgr2=0x7f9a8c0b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b5950 0x7f9a8c0b7cc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009e50 tx=0x7f9a8402a710 comp rx=0 tx=0).stop 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 -- 172.21.15.146:0/745455941 shutdown_connections 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b5950 0x7f9a8c0b7cc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 --2- 172.21.15.146:0/745455941 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c0b6610 0x7f9a8c0d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 -- 172.21.15.146:0/745455941 >> 172.21.15.146:0/745455941 conn(0x7f9a8c0d0240 msgr2=0x7f9a8c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:36.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.815+0000 7f9a91384640 1 -- 172.21.15.146:0/745455941 shutdown_connections 2024-04-25T15:22:36.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.816+0000 7f9a91384640 1 -- 172.21.15.146:0/745455941 wait complete. 2024-04-25T15:22:36.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.817+0000 7f9a91384640 1 Processor -- start 2024-04-25T15:22:36.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.817+0000 7f9a91384640 1 -- start start 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.818+0000 7f9a91384640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 0x7f9a8c1f9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.818+0000 7f9a91384640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c15ffa0 0x7f9a8c15d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.818+0000 7f9a91384640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a8c162ca0 0x7f9a8c1630b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.818+0000 7f9a91384640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c0d1da0 con 0x7f9a8c0b6610 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.818+0000 7f9a91384640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1603b0 con 0x7f9a8c162ca0 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.818+0000 7f9a91384640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c0d1f20 con 0x7f9a8c15ffa0 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 0x7f9a8c1f9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 0x7f9a8c1f9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50872/0 (socket says 172.21.15.146:50872) 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 -- 172.21.15.146:0/3706344945 learned_addr learned my addr 172.21.15.146:0/3706344945 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8b7fe640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a8c162ca0 0x7f9a8c1630b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a83fff640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c15ffa0 0x7f9a8c15d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 -- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c15ffa0 msgr2=0x7f9a8c15d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c15ffa0 0x7f9a8c15d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 -- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a8c162ca0 msgr2=0x7f9a8c1630b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a8c162ca0 0x7f9a8c1630b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 -- 172.21.15.146:0/3706344945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c1635f0 con 0x7f9a8c0b6610 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a83fff640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c15ffa0 0x7f9a8c15d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a8affd640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 0x7f9a8c1f9550 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00d800 tx=0x7f9a7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c004030 con 0x7f9a8c0b6610 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7c0041d0 con 0x7f9a8c0b6610 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00b6c0 con 0x7f9a8c0b6610 2024-04-25T15:22:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c1f9a90 con 0x7f9a8c0b6610 2024-04-25T15:22:36.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.819+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c1fa280 con 0x7f9a8c0b6610 2024-04-25T15:22:36.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.820+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c06a0e0 con 0x7f9a8c0b6610 2024-04-25T15:22:36.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.821+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f9a7c00b860 con 0x7f9a8c0b6610 2024-04-25T15:22:36.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.821+0000 7f9a837fe640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a6c0745d0 0x7f9a6c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:36.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.821+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f9a7c095740 con 0x7f9a8c0b6610 2024-04-25T15:22:36.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.821+0000 7f9a83fff640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a6c0745d0 0x7f9a6c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:36.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.822+0000 7f9a83fff640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a6c0745d0 0x7f9a6c076ae0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0c6a70 tx=0x7f9a74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:36.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:36.826+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a8c1635f0 con 0x7f9a8c0b6610 2024-04-25T15:22:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.003+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a8c0d2bd0 con 0x7f9a8c0b6610 2024-04-25T15:22:37.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.004+0000 7f9a837fe640 1 -- 172.21.15.146:0/3706344945 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+17029 (secure 0 0 0) 0x7f9a7c05fd20 con 0x7f9a8c0b6610 2024-04-25T15:22:37.006 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:37.006 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":56,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:28.267703+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a6c0745d0 msgr2=0x7f9a6c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a6c0745d0 0x7f9a6c076ae0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0c6a70 tx=0x7f9a74009040 comp rx=0 tx=0).stop 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 msgr2=0x7f9a8c1f9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 0x7f9a8c1f9550 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00d800 tx=0x7f9a7c00dcc0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 shutdown_connections 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9a6c0745d0 0x7f9a6c076ae0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a8c162ca0 0x7f9a8c1630b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a8c15ffa0 0x7f9a8c15d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 --2- 172.21.15.146:0/3706344945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0b6610 0x7f9a8c1f9550 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 >> 172.21.15.146:0/3706344945 conn(0x7f9a8c0d0240 msgr2=0x7f9a8c070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 shutdown_connections 2024-04-25T15:22:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.006+0000 7f9a91384640 1 -- 172.21.15.146:0/3706344945 wait complete. 2024-04-25T15:22:37.019 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:37.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.129+0000 7fd38394d640 1 Processor -- start 2024-04-25T15:22:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.129+0000 7fd38394d640 1 -- start start 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.130+0000 7fd38394d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 0x7fd37c09bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd38394d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd37c0b2400 0x7fd37c0b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd38394d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37c0b1960 con 0x7fd37c0ccd40 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd38394d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fd37c0b3ce0 con 0x7fd37c0b2400 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd38394d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd37c0b1c00 con 0x7fd37c0d2d90 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd37c0b2400 0x7fd37c0b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd380ee8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 0x7fd37c09bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd37c0b2400 0x7fd37c0b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56374/0 (socket says 172.21.15.146:56374) 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd3816e9640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fd37c0ccd40 0x7fd37c0d2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:49476/0 (socket says 172.21.15.146:49476) 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd380ee8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 0x7fd37c09bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:38686/0 (socket says 172.21.15.146:38686) 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 -- 172.21.15.146:0/3839841861 learned_addr learned my addr 172.21.15.146:0/3839841861 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 -- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 msgr2=0x7fd37c09bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 --2- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 0x7fd37c09bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 -- 172.21.15.146:0/3839841861 >> v1:172.21.15.177:6789/0 conn(0x7fd37c0ccd40 legacy=0x7fd37c0d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd381eea640 1 -- 172.21.15.146:0/3839841861 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37c0b1ee0 con 0x7fd37c0b2400 2024-04-25T15:22:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.131+0000 7fd380ee8640 1 --2- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 0x7fd37c09bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.132+0000 7fd381eea640 1 --2- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0b2400 0x7fd37c0b2810 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd374009e50 tx=0x7fd37402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e50f9f4b3530dc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.132+0000 7fd373fff640 1 -- 172.21.15.146:0/3839841861 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374036070 con 0x7fd37c0b2400 2024-04-25T15:22:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.132+0000 7fd373fff640 1 -- 172.21.15.146:0/3839841861 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd374031600 con 0x7fd37c0b2400 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.132+0000 7fd373fff640 1 -- 172.21.15.146:0/3839841861 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740319d0 con 0x7fd37c0b2400 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 -- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0b2400 msgr2=0x7fd37c0b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 --2- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0b2400 0x7fd37c0b2810 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd374009e50 tx=0x7fd37402a710 comp rx=0 tx=0).stop 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 -- 172.21.15.146:0/3839841861 shutdown_connections 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 --2- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0b2400 0x7fd37c0b2810 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 --2- 172.21.15.146:0/3839841861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0d2d90 0x7fd37c09bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 -- 172.21.15.146:0/3839841861 >> 172.21.15.146:0/3839841861 conn(0x7fd37c0b7400 msgr2=0x7fd37c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 -- 172.21.15.146:0/3839841861 shutdown_connections 2024-04-25T15:22:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.133+0000 7fd38394d640 1 -- 172.21.15.146:0/3839841861 wait complete. 2024-04-25T15:22:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.135+0000 7fd38394d640 1 Processor -- start 2024-04-25T15:22:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.135+0000 7fd38394d640 1 -- start start 2024-04-25T15:22:37.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd38394d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0b2400 0x7fd37c0ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd38394d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0ccd40 0x7fd37c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd38394d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 0x7fd37c152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd38394d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd37c0b3ac0 con 0x7fd37c0710b0 2024-04-25T15:22:37.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd38394d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd37c0b3940 con 0x7fd37c0ccd40 2024-04-25T15:22:37.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd38394d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd37c09b440 con 0x7fd37c0b2400 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd381eea640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 0x7fd37c152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd381eea640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 0x7fd37c152780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50880/0 (socket says 172.21.15.146:50880) 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd381eea640 1 -- 172.21.15.146:0/3299522089 learned_addr learned my addr 172.21.15.146:0/3299522089 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd380ee8640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0ccd40 0x7fd37c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.136+0000 7fd3816e9640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0b2400 0x7fd37c0ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd381eea640 1 -- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0b2400 msgr2=0x7fd37c0ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd381eea640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0b2400 0x7fd37c0ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd381eea640 1 -- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0ccd40 msgr2=0x7fd37c070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd381eea640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0ccd40 0x7fd37c070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd381eea640 1 -- 172.21.15.146:0/3299522089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37c0cb1a0 con 0x7fd37c0710b0 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd381eea640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 0x7fd37c152780 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd374009f70 tx=0x7fd374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37403f020 con 0x7fd37c0710b0 2024-04-25T15:22:37.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd374037040 con 0x7fd37c0710b0 2024-04-25T15:22:37.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd37c0cb480 con 0x7fd37c0710b0 2024-04-25T15:22:37.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374036070 con 0x7fd37c0710b0 2024-04-25T15:22:37.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.137+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd37c160970 con 0x7fd37c0710b0 2024-04-25T15:22:37.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.138+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd37c06a0e0 con 0x7fd37c0710b0 2024-04-25T15:22:37.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.139+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fd374042020 con 0x7fd37c0710b0 2024-04-25T15:22:37.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.139+0000 7fd371ffb640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3600745d0 0x7fd360076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.139+0000 7fd3816e9640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3600745d0 0x7fd360076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.139+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fd3740b5910 con 0x7fd37c0710b0 2024-04-25T15:22:37.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.140+0000 7fd3816e9640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3600745d0 0x7fd360076ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd378000fb0 tx=0x7fd378007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.144+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd374049030 con 0x7fd37c0710b0 2024-04-25T15:22:37.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.317+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd37c0caf30 con 0x7fd37c0710b0 2024-04-25T15:22:37.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.318+0000 7fd371ffb640 1 -- 172.21.15.146:0/3299522089 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7fd37407f700 con 0x7fd37c0710b0 2024-04-25T15:22:37.320 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3600745d0 msgr2=0x7fd360076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3600745d0 0x7fd360076ae0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd378000fb0 tx=0x7fd378007e70 comp rx=0 tx=0).stop 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 msgr2=0x7fd37c152780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 0x7fd37c152780 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd374009f70 tx=0x7fd374004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 shutdown_connections 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3600745d0 0x7fd360076ae0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c0710b0 0x7fd37c152780 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd37c0ccd40 0x7fd37c070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 --2- 172.21.15.146:0/3299522089 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd37c0b2400 0x7fd37c0ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 >> 172.21.15.146:0/3299522089 conn(0x7fd37c0b7400 msgr2=0x7fd37c0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 shutdown_connections 2024-04-25T15:22:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.320+0000 7fd38394d640 1 -- 172.21.15.146:0/3299522089 wait complete. 2024-04-25T15:22:37.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:37.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.442+0000 7fe8952ca640 1 Processor -- start 2024-04-25T15:22:37.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.442+0000 7fe8952ca640 1 -- start start 2024-04-25T15:22:37.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe8952ca640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8900b3940 0x7fe8900d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe8952ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe8900d2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe8952ca640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8900b2d90 con 0x7fe8900ce650 2024-04-25T15:22:37.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe8952ca640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fe8900b2b60 con 0x7fe8900b3940 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe8952ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe8900b3050 con 0x7fe8900b3d50 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe88effd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8900b3940 0x7fe8900d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe88e7fc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe8900d2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe88effd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8900b3940 0x7fe8900d2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56394/0 (socket says 172.21.15.146:56394) 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe88e7fc640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe8900d2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:38704/0 (socket says 172.21.15.146:38704) 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe88f7fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe8900ce650 0x7fe8900cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:49480/0 (socket says 172.21.15.146:49480) 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.443+0000 7fe88effd640 1 -- 172.21.15.146:0/3431766437 learned_addr learned my addr 172.21.15.146:0/3431766437 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.444+0000 7fe88effd640 1 -- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 msgr2=0x7fe8900d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.444+0000 7fe88effd640 1 --2- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe8900d2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.444+0000 7fe88effd640 1 -- 172.21.15.146:0/3431766437 >> v1:172.21.15.177:6789/0 conn(0x7fe8900ce650 legacy=0x7fe8900cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.444+0000 7fe88effd640 1 -- 172.21.15.146:0/3431766437 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8900b3330 con 0x7fe8900b3940 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.444+0000 7fe88dffb640 1 -- 172.21.15.146:0/3431766437 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2183263073 0 0) 0x7fe8900b2d90 con 0x7fe8900ce650 2024-04-25T15:22:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.444+0000 7fe88e7fc640 1 --2- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe8900d2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:37.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe88effd640 1 --2- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900d2780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe880009e50 tx=0x7fe88002a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c43d93943c54be90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe88dffb640 1 -- 172.21.15.146:0/3431766437 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880036070 con 0x7fe8900b3940 2024-04-25T15:22:37.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe88dffb640 1 -- 172.21.15.146:0/3431766437 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe880031600 con 0x7fe8900b3940 2024-04-25T15:22:37.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe88dffb640 1 -- 172.21.15.146:0/3431766437 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8800319d0 con 0x7fe8900b3940 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe8952ca640 1 -- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 msgr2=0x7fe8900d2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe8952ca640 1 --2- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900d2780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe880009e50 tx=0x7fe88002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe8952ca640 1 -- 172.21.15.146:0/3431766437 shutdown_connections 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe8952ca640 1 --2- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe8900d2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe8952ca640 1 --2- 172.21.15.146:0/3431766437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900d2780 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.445+0000 7fe8952ca640 1 -- 172.21.15.146:0/3431766437 >> 172.21.15.146:0/3431766437 conn(0x7fe8900b7490 msgr2=0x7fe8900b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.446+0000 7fe8952ca640 1 -- 172.21.15.146:0/3431766437 shutdown_connections 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.446+0000 7fe8952ca640 1 -- 172.21.15.146:0/3431766437 wait complete. 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.447+0000 7fe8952ca640 1 Processor -- start 2024-04-25T15:22:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.447+0000 7fe8952ca640 1 -- start start 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe8952ca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe8952ca640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe890153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe8952ca640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 0x7fe8900b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe8952ca640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe89009b7b0 con 0x7fe8900ce650 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe8952ca640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe89009b630 con 0x7fe8900b3940 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe8952ca640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fe8900cf170 con 0x7fe8900b3d50 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 0x7fe8900b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 0x7fe8900b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50886/0 (socket says 172.21.15.146:50886) 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 -- 172.21.15.146:0/1906298992 learned_addr learned my addr 172.21.15.146:0/1906298992 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88e7fc640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe890153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88effd640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 -- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 msgr2=0x7fe890153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe890153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 -- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 msgr2=0x7fe8900c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 -- 172.21.15.146:0/1906298992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8900c8ff0 con 0x7fe8900ce650 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.449+0000 7fe88f7fe640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 0x7fe8900b0d90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c440 tx=0x7fe88800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.450+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8880074e0 con 0x7fe8900ce650 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.450+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8880047e0 con 0x7fe8900ce650 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.450+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8900ca2b0 con 0x7fe8900ce650 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.450+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888004bd0 con 0x7fe8900ce650 2024-04-25T15:22:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.450+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8900ca720 con 0x7fe8900ce650 2024-04-25T15:22:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.451+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe89006a0e0 con 0x7fe8900ce650 2024-04-25T15:22:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.451+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fe888011070 con 0x7fe8900ce650 2024-04-25T15:22:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.452+0000 7fe8777fe640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe87c0745d0 0x7fe87c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.452+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fe888094f00 con 0x7fe8900ce650 2024-04-25T15:22:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.452+0000 7fe88effd640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe87c0745d0 0x7fe87c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.452+0000 7fe88effd640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe87c0745d0 0x7fe87c076ae0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe880033000 tx=0x7fe880009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.456+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8900c8ff0 con 0x7fe8900ce650 2024-04-25T15:22:37.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.630+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe8900ca480 con 0x7fe8900ce650 2024-04-25T15:22:37.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.632+0000 7fe8777fe640 1 -- 172.21.15.146:0/1906298992 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7fe88805ed70 con 0x7fe8900ce650 2024-04-25T15:22:37.634 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:37.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe87c0745d0 msgr2=0x7fe87c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe87c0745d0 0x7fe87c076ae0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe880033000 tx=0x7fe880009000 comp rx=0 tx=0).stop 2024-04-25T15:22:37.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 msgr2=0x7fe8900b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 0x7fe8900b0d90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c440 tx=0x7fe88800c900 comp rx=0 tx=0).stop 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 shutdown_connections 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fe87c0745d0 0x7fe87c076ae0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8900ce650 0x7fe8900b0d90 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8900b3d50 0x7fe890153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 --2- 172.21.15.146:0/1906298992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8900b3940 0x7fe8900c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 >> 172.21.15.146:0/1906298992 conn(0x7fe8900b7490 msgr2=0x7fe89006d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 shutdown_connections 2024-04-25T15:22:37.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.634+0000 7fe8952ca640 1 -- 172.21.15.146:0/1906298992 wait complete. 2024-04-25T15:22:37.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:37.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.756+0000 7f5e53900640 1 Processor -- start 2024-04-25T15:22:37.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.757+0000 7f5e53900640 1 -- start start 2024-04-25T15:22:37.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e53900640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0d2d90 0x7f5e4c09bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e53900640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e4c0b2400 0x7f5e4c0b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e53900640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e4c0b1960 con 0x7f5e4c0ccd40 2024-04-25T15:22:37.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e53900640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5e4c0b3ce0 con 0x7f5e4c0b2400 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e53900640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c0b1c00 con 0x7f5e4c0d2d90 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e51e9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e4c0b2400 0x7f5e4c0b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e5169c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5e4c0ccd40 0x7f5e4c0d2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32874/0 (socket says 172.21.15.146:32874) 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e51e9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e4c0b2400 0x7f5e4c0b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50892/0 (socket says 172.21.15.146:50892) 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e5169c640 1 -- 172.21.15.146:0/644175578 learned_addr learned my addr 172.21.15.146:0/644175578 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e50e9b640 1 --2- 172.21.15.146:0/644175578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0d2d90 0x7f5e4c09bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e51e9d640 1 -- 172.21.15.146:0/644175578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0d2d90 msgr2=0x7f5e4c09bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e51e9d640 1 --2- 172.21.15.146:0/644175578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0d2d90 0x7f5e4c09bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e51e9d640 1 -- 172.21.15.146:0/644175578 >> v1:172.21.15.146:6789/0 conn(0x7f5e4c0ccd40 legacy=0x7f5e4c0d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e43fff640 1 -- 172.21.15.146:0/644175578 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670390630 0 0) 0x7f5e4c0b1960 con 0x7f5e4c0ccd40 2024-04-25T15:22:37.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.758+0000 7f5e51e9d640 1 -- 172.21.15.146:0/644175578 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c0b1ee0 con 0x7f5e4c0b2400 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.759+0000 7f5e51e9d640 1 --2- 172.21.15.146:0/644175578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c0b2400 0x7f5e4c0b2810 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009d90 tx=0x7f5e4402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13166e1a7ed3b864 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.759+0000 7f5e43fff640 1 -- 172.21.15.146:0/644175578 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44007740 con 0x7f5e4c0b2400 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.759+0000 7f5e43fff640 1 -- 172.21.15.146:0/644175578 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e440078e0 con 0x7f5e4c0b2400 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.759+0000 7f5e43fff640 1 -- 172.21.15.146:0/644175578 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44007cb0 con 0x7f5e4c0b2400 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 -- 172.21.15.146:0/644175578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c0b2400 msgr2=0x7f5e4c0b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 --2- 172.21.15.146:0/644175578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c0b2400 0x7f5e4c0b2810 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009d90 tx=0x7f5e4402a660 comp rx=0 tx=0).stop 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 -- 172.21.15.146:0/644175578 shutdown_connections 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 --2- 172.21.15.146:0/644175578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c0b2400 0x7f5e4c0b2810 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 --2- 172.21.15.146:0/644175578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0d2d90 0x7f5e4c09bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 -- 172.21.15.146:0/644175578 >> 172.21.15.146:0/644175578 conn(0x7f5e4c0b7400 msgr2=0x7f5e4c0b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 -- 172.21.15.146:0/644175578 shutdown_connections 2024-04-25T15:22:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.760+0000 7f5e53900640 1 -- 172.21.15.146:0/644175578 wait complete. 2024-04-25T15:22:37.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.762+0000 7f5e53900640 1 Processor -- start 2024-04-25T15:22:37.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.762+0000 7f5e53900640 1 -- start start 2024-04-25T15:22:37.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e53900640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0b2400 0x7f5e4c0ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e53900640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e4c0ccd40 0x7f5e4c0c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e53900640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 0x7f5e4c0c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e53900640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c0b3ac0 con 0x7f5e4c14d070 2024-04-25T15:22:37.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e53900640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c0b3940 con 0x7f5e4c0ccd40 2024-04-25T15:22:37.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e53900640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4c09b440 con 0x7f5e4c0b2400 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e51e9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 0x7f5e4c0c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e51e9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 0x7f5e4c0c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50908/0 (socket says 172.21.15.146:50908) 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e51e9d640 1 -- 172.21.15.146:0/1832239951 learned_addr learned my addr 172.21.15.146:0/1832239951 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e5169c640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0b2400 0x7f5e4c0ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.763+0000 7f5e50e9b640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e4c0ccd40 0x7f5e4c0c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e51e9d640 1 -- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0b2400 msgr2=0x7f5e4c0ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e51e9d640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0b2400 0x7f5e4c0ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e51e9d640 1 -- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e4c0ccd40 msgr2=0x7f5e4c0c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e51e9d640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e4c0ccd40 0x7f5e4c0c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e51e9d640 1 -- 172.21.15.146:0/1832239951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c0ceab0 con 0x7f5e4c14d070 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e51e9d640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 0x7f5e4c0c7d10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009eb0 tx=0x7f5e44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4403f070 con 0x7f5e4c14d070 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e44031c40 con 0x7f5e4c14d070 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4c0c8430 con 0x7f5e4c14d070 2024-04-25T15:22:37.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44003990 con 0x7f5e4c14d070 2024-04-25T15:22:37.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.764+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4c0c89f0 con 0x7f5e4c14d070 2024-04-25T15:22:37.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.765+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e4c06a0e0 con 0x7f5e4c14d070 2024-04-25T15:22:37.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.766+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f5e44003b30 con 0x7f5e4c14d070 2024-04-25T15:22:37.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.766+0000 7f5e41ffb640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e300745d0 0x7f5e30076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:37.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.766+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f5e440b6810 con 0x7f5e4c14d070 2024-04-25T15:22:37.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.766+0000 7f5e5169c640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e300745d0 0x7f5e30076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:37.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.767+0000 7f5e5169c640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e300745d0 0x7f5e30076ae0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5e48001000 tx=0x7f5e48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:37.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.771+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e4c0ceab0 con 0x7f5e4c14d070 2024-04-25T15:22:37.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.944+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5e4c0d05d0 con 0x7f5e4c14d070 2024-04-25T15:22:37.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.944+0000 7f5e41ffb640 1 -- 172.21.15.146:0/1832239951 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f5e4c0d05d0 con 0x7f5e4c14d070 2024-04-25T15:22:37.946 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:37.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e300745d0 msgr2=0x7f5e30076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e300745d0 0x7f5e30076ae0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5e48001000 tx=0x7f5e48009040 comp rx=0 tx=0).stop 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 msgr2=0x7f5e4c0c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 0x7f5e4c0c7d10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009eb0 tx=0x7f5e44004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 shutdown_connections 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5e300745d0 0x7f5e30076ae0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e4c14d070 0x7f5e4c0c7d10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e4c0ccd40 0x7f5e4c0c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 --2- 172.21.15.146:0/1832239951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e4c0b2400 0x7f5e4c0ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.946+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 >> 172.21.15.146:0/1832239951 conn(0x7f5e4c0b7400 msgr2=0x7f5e4c0af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.947+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 shutdown_connections 2024-04-25T15:22:37.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:37.947+0000 7f5e53900640 1 -- 172.21.15.146:0/1832239951 wait complete. 2024-04-25T15:22:37.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:38.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.068+0000 7f816977a640 1 Processor -- start 2024-04-25T15:22:38.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.068+0000 7f816977a640 1 -- start start 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f816977a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f81640d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f816977a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f81640d0870 0x7f81640c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f816977a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81640cfa90 con 0x7f816406f120 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f816977a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f81640d0050 con 0x7f81640d0870 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f816977a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f816409bfb0 con 0x7f816409b2b0 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f8162ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f816406f120 0x7f81640b3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32876/0 (socket says 172.21.15.146:32876) 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f8162ffd640 1 -- 172.21.15.146:0/3263259660 learned_addr learned my addr 172.21.15.146:0/3263259660 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f81637fe640 1 --2- 172.21.15.146:0/3263259660 >> v2:172.21.15.177:3300/0 conn(0x7f81640d0870 0x7f81640c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.069+0000 7f815bfff640 1 --2- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f81640d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.070+0000 7f81627fc640 1 -- 172.21.15.146:0/3263259660 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2479343141 0 0) 0x7f81640cfa90 con 0x7f816406f120 2024-04-25T15:22:38.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.070+0000 7f81627fc640 1 -- 172.21.15.146:0/3263259660 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8150003520 con 0x7f816406f120 2024-04-25T15:22:38.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.070+0000 7f81637fe640 1 -- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 msgr2=0x7f81640d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:38.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.070+0000 7f81637fe640 1 --2- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f81640d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.070+0000 7f81637fe640 1 -- 172.21.15.146:0/3263259660 >> v1:172.21.15.146:6789/0 conn(0x7f816406f120 legacy=0x7f81640b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.070+0000 7f81637fe640 1 -- 172.21.15.146:0/3263259660 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81640cfd70 con 0x7f81640d0870 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.071+0000 7f81637fe640 1 --2- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81640d0870 0x7f81640c8ab0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f815c009d90 tx=0x7f815c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13c20685ff6ccaa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.071+0000 7f81627fc640 1 -- 172.21.15.146:0/3263259660 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c031600 con 0x7f81640d0870 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.071+0000 7f81627fc640 1 -- 172.21.15.146:0/3263259660 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f815c0317a0 con 0x7f81640d0870 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.071+0000 7f81627fc640 1 -- 172.21.15.146:0/3263259660 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c031b50 con 0x7f81640d0870 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.071+0000 7f816977a640 1 -- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81640d0870 msgr2=0x7f81640c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.071+0000 7f816977a640 1 --2- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81640d0870 0x7f81640c8ab0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f815c009d90 tx=0x7f815c02a5d0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.072+0000 7f816977a640 1 -- 172.21.15.146:0/3263259660 shutdown_connections 2024-04-25T15:22:38.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.072+0000 7f816977a640 1 --2- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81640d0870 0x7f81640c8ab0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.072+0000 7f816977a640 1 --2- 172.21.15.146:0/3263259660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f81640d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.072+0000 7f816977a640 1 -- 172.21.15.146:0/3263259660 >> 172.21.15.146:0/3263259660 conn(0x7f81640b7530 msgr2=0x7f81640b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:38.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.072+0000 7f816977a640 1 -- 172.21.15.146:0/3263259660 shutdown_connections 2024-04-25T15:22:38.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.072+0000 7f816977a640 1 -- 172.21.15.146:0/3263259660 wait complete. 2024-04-25T15:22:38.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.073+0000 7f816977a640 1 Processor -- start 2024-04-25T15:22:38.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.073+0000 7f816977a640 1 -- start start 2024-04-25T15:22:38.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f816406f120 0x7f8164162690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f816415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 0x7f8164071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81640b4120 con 0x7f81640d0870 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81640d0c80 con 0x7f816406f120 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f816409b6c0 con 0x7f816409b2b0 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 0x7f8164071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 0x7f8164071740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50922/0 (socket says 172.21.15.146:50922) 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 -- 172.21.15.146:0/54258430 learned_addr learned my addr 172.21.15.146:0/54258430 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f815bfff640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f816415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f8162ffd640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f816406f120 0x7f8164162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 -- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 msgr2=0x7f816415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f816415d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 -- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f816406f120 msgr2=0x7f8164162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f816406f120 0x7f8164162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 -- 172.21.15.146:0/54258430 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8164163070 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f81637fe640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 0x7f8164071740 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f815c02abc0 tx=0x7f815c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c03f070 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f815c03b840 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8164160590 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c03bc30 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.075+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8164160b50 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.076+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8164163350 con 0x7f81640d0870 2024-04-25T15:22:38.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.077+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f815c04c3f0 con 0x7f81640d0870 2024-04-25T15:22:38.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.077+0000 7f815b7fe640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81440745d0 0x7f8144076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.078+0000 7f8162ffd640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81440745d0 0x7f8144076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.078+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f815c0b6700 con 0x7f81640d0870 2024-04-25T15:22:38.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.078+0000 7f8162ffd640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81440745d0 0x7f8144076ae0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8154005880 tx=0x7f8154009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:38.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.082+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f815c080460 con 0x7f81640d0870 2024-04-25T15:22:38.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.269+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f816406a0e0 con 0x7f81640d0870 2024-04-25T15:22:38.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.269+0000 7f815b7fe640 1 -- 172.21.15.146:0/54258430 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f815c084760 con 0x7f81640d0870 2024-04-25T15:22:38.271 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-04-25T15:22:38.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81440745d0 msgr2=0x7f8144076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81440745d0 0x7f8144076ae0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8154005880 tx=0x7f8154009040 comp rx=0 tx=0).stop 2024-04-25T15:22:38.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 msgr2=0x7f8164071740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 0x7f8164071740 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f815c02abc0 tx=0x7f815c004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:38.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 shutdown_connections 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f81440745d0 0x7f8144076ae0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81640d0870 0x7f8164071740 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f816409b2b0 0x7f816415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 --2- 172.21.15.146:0/54258430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f816406f120 0x7f8164162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 >> 172.21.15.146:0/54258430 conn(0x7f81640b7530 msgr2=0x7f81640b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 shutdown_connections 2024-04-25T15:22:38.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.271+0000 7f816977a640 1 -- 172.21.15.146:0/54258430 wait complete. 2024-04-25T15:22:38.285 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:22:38.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:38.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[43864]: fuse finished with error 0 and tester_r 0 2024-04-25T15:22:38.314 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:22:38.386 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:22:38.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.387 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:22:38.404 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:38.404 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:38.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1449 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:22:38.421 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi146.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-04-25T15:22:38.421 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:22:38.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.421 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T15:22:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:22:38.476 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-25T15:22:38.476 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:38.476 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:38.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3659 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:22:38.494 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi177.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-04-25T15:22:38.494 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:22:38.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.494 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T15:22:38.551 INFO:teuthology.orchestra.run.smithi177.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-25T15:22:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:22:38.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:22:38.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.660+0000 7fb325642640 1 Processor -- start 2024-04-25T15:22:38.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.660+0000 7fb325642640 1 -- start start 2024-04-25T15:22:38.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb325642640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb3200d4f60 0x7fb3200d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb325642640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb325642640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3200b9200 con 0x7fb3200d0020 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb325642640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb32006c550 con 0x7fb3200d4f60 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb325642640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb32006c830 con 0x7fb32006ccc0 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31effd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb3200d4f60 0x7fb3200d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32006d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:38766/0 (socket says 172.21.15.146:38766) 2024-04-25T15:22:38.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 -- 172.21.15.146:0/65545515 learned_addr learned my addr 172.21.15.146:0/65545515 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 -- 172.21.15.146:0/65545515 >> v2:172.21.15.177:3300/0 conn(0x7fb3200d4f60 msgr2=0x7fb3200d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 --2- 172.21.15.146:0/65545515 >> v2:172.21.15.177:3300/0 conn(0x7fb3200d4f60 0x7fb3200d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 -- 172.21.15.146:0/65545515 >> v1:172.21.15.177:6789/0 conn(0x7fb3200d0020 legacy=0x7fb3200d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31f7fe640 1 -- 172.21.15.146:0/65545515 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3200d1070 con 0x7fb32006ccc0 2024-04-25T15:22:38.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.662+0000 7fb31effd640 1 --2- 172.21.15.146:0/65545515 >> v2:172.21.15.177:3300/0 conn(0x7fb3200d4f60 0x7fb3200d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.663+0000 7fb31f7fe640 1 --2- 172.21.15.146:0/65545515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32006d0d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb318009e50 tx=0x7fb31802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99185090b243e57f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.663+0000 7fb31dffb640 1 -- 172.21.15.146:0/65545515 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318036070 con 0x7fb32006ccc0 2024-04-25T15:22:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.663+0000 7fb31dffb640 1 -- 172.21.15.146:0/65545515 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb318031600 con 0x7fb32006ccc0 2024-04-25T15:22:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.663+0000 7fb31dffb640 1 -- 172.21.15.146:0/65545515 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3180319d0 con 0x7fb32006ccc0 2024-04-25T15:22:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 -- 172.21.15.146:0/65545515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 msgr2=0x7fb32006d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 --2- 172.21.15.146:0/65545515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32006d0d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb318009e50 tx=0x7fb31802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 -- 172.21.15.146:0/65545515 shutdown_connections 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 --2- 172.21.15.146:0/65545515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32006d0d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 --2- 172.21.15.146:0/65545515 >> v2:172.21.15.177:3300/0 conn(0x7fb3200d4f60 0x7fb3200d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 -- 172.21.15.146:0/65545515 >> 172.21.15.146:0/65545515 conn(0x7fb3200d2450 msgr2=0x7fb3200d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 -- 172.21.15.146:0/65545515 shutdown_connections 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.664+0000 7fb325642640 1 -- 172.21.15.146:0/65545515 wait complete. 2024-04-25T15:22:38.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.666+0000 7fb325642640 1 Processor -- start 2024-04-25T15:22:38.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.666+0000 7fb325642640 1 -- start start 2024-04-25T15:22:38.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb325642640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb325642640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 0x7fb320150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb325642640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3200d4f60 0x7fb32015ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb325642640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb3200b5150 con 0x7fb3200d0020 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb325642640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3200d15f0 con 0x7fb3200d4f60 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb325642640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb3200b52d0 con 0x7fb32006ccc0 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.667+0000 7fb31e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 0x7fb320150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 0x7fb320150940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50932/0 (socket says 172.21.15.146:50932) 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 -- 172.21.15.146:0/2533099748 learned_addr learned my addr 172.21.15.146:0/2533099748 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31effd640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31f7fe640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3200d4f60 0x7fb32015ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 -- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 msgr2=0x7fb32015c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32015c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 -- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3200d4f60 msgr2=0x7fb32015ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3200d4f60 0x7fb32015ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 -- 172.21.15.146:0/2533099748 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3201f6880 con 0x7fb3200d0020 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb31e7fc640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 0x7fb320150940 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb31400c3f0 tx=0x7fb31400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3140052d0 con 0x7fb3200d0020 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb314005490 con 0x7fb3200d0020 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3201f6db0 con 0x7fb3200d0020 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314007260 con 0x7fb3200d0020 2024-04-25T15:22:38.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.668+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3201f7370 con 0x7fb3200d0020 2024-04-25T15:22:38.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.669+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3201f6b20 con 0x7fb3200d0020 2024-04-25T15:22:38.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.670+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fb314007400 con 0x7fb3200d0020 2024-04-25T15:22:38.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.670+0000 7fb3077fe640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb30c0745d0 0x7fb30c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:38.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.671+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fb3140948d0 con 0x7fb3200d0020 2024-04-25T15:22:38.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.671+0000 7fb31effd640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb30c0745d0 0x7fb30c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:38.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.671+0000 7fb31effd640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb30c0745d0 0x7fb30c076ae0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb310009560 tx=0x7fb310009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:38.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.675+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3200b9200 con 0x7fb3200d0020 2024-04-25T15:22:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.885+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb32006a0e0 con 0x7fb3200d0020 2024-04-25T15:22:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.886+0000 7fb3077fe640 1 -- 172.21.15.146:0/2533099748 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7fb31405e750 con 0x7fb3200d0020 2024-04-25T15:22:38.887 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb30c0745d0 msgr2=0x7fb30c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb30c0745d0 0x7fb30c076ae0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb310009560 tx=0x7fb310009190 comp rx=0 tx=0).stop 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 msgr2=0x7fb320150940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 0x7fb320150940 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb31400c3f0 tx=0x7fb31400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 shutdown_connections 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb30c0745d0 0x7fb30c076ae0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3200d4f60 0x7fb32015ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200d0020 0x7fb320150940 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 --2- 172.21.15.146:0/2533099748 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32006ccc0 0x7fb32015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 >> 172.21.15.146:0/2533099748 conn(0x7fb3200d2450 msgr2=0x7fb3200d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 shutdown_connections 2024-04-25T15:22:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:38.888+0000 7fb325642640 1 -- 172.21.15.146:0/2533099748 wait complete. 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:22:38.901 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:22:38.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.902 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:38.917 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:38.918 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7820:1ff:feb4:c7f1/64 scope link 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:38.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.920 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:22:38.980 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:22:38.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:38.981 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:22:39.036 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:22:39.036 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:22:39.037 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:22:39.037 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:22:39.037 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:22:39.037 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:22:39.037 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:22:39.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:22:39.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:22:39.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:22:39.037 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:22:39.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:22:39.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:39.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:39.160 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:22:39.175 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:22:39.175 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:22:39.175 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:22:39.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:39.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-25T15:22:39.239 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T15:22:39.239 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:39.239 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:39.294 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:22:39.295 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:39.295 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:39.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:22:39.413 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:39.469 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:22:39.469 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:22:39.469 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:22:39.469 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:22:39.469 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:22:39.470 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:22:39.471 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:39.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:39.472 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:22:39.524 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:22:39.524 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb) 2024-04-25T15:22:39.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:22:39.590 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:39.648 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:22:39.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:22:39.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:39.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:39.651 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:22:39.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:2024-04-25T15:22:39.654+0000 7f5f111ff480 -1 init, newargv = 0x5637e2266390 newargc=15 2024-04-25T15:22:39.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[59361]: starting ceph client 2024-04-25T15:22:39.664 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[59361]: starting fuse 2024-04-25T15:22:39.671 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:22:39.671 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:22:39.672 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.0")) 2024-04-25T15:22:39.673 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:22:39.788 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.0.59361.asok 2024-04-25T15:22:39.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:39.798 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59361.asok 2024-04-25T15:22:39.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:39.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59361.asok status 2024-04-25T15:22:39.939 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "0", 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "59361", 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb" 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6753, 2024-04-25T15:22:39.940 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 6753 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3679804753 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3679804753 2024-04-25T15:22:39.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.6753 192.168.0.1:0/3679804753", 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/3679804753", 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 5, 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 30, 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 56, 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:22:39.942 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:39.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:22:40.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.055+0000 7f23f5d46640 1 Processor -- start 2024-04-25T15:22:40.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.055+0000 7f23f5d46640 1 -- start start 2024-04-25T15:22:40.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.056+0000 7f23f5d46640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 0x7f23f006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.056+0000 7f23f5d46640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f23f0070190 0x7f23f0169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23f5d46640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23f00705a0 con 0x7f23f00b7350 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23f5d46640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f23f0071bc0 con 0x7f23f0070190 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23f5d46640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f23f00d1b00 con 0x7f23f00715e0 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ef7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f23f00b7350 0x7f23f0160000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:32886/0 (socket says 172.21.15.146:32886) 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ef7fe640 1 -- 172.21.15.146:0/1505082493 learned_addr learned my addr 172.21.15.146:0/1505082493 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 0x7f23f006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23effff640 1 --2- 172.21.15.146:0/1505082493 >> v2:172.21.15.177:3300/0 conn(0x7f23f0070190 0x7f23f0169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 535413003 0 0) 0x7f23e0001110 con 0x7f23f00b7350 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23d8003520 con 0x7f23f00b7350 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2602370530 0 0) 0x7f23d8003520 con 0x7f23f00b7350 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 msgr2=0x7f23f006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:40.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 0x7f23f006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 >> v2:172.21.15.177:3300/0 conn(0x7f23f0070190 msgr2=0x7f23f0169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 --2- 172.21.15.146:0/1505082493 >> v2:172.21.15.177:3300/0 conn(0x7f23f0070190 0x7f23f0169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f00d1de0 con 0x7f23f00b7350 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f23e0001840 con 0x7f23f00b7350 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 >> v1:172.21.15.146:6789/0 conn(0x7f23f00b7350 legacy=0x7f23f0160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 0x7f23d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23d8003f30 0x7f23d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8004d40 0x7f23d8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23f0071bc0 con 0x7f23d8003f30 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23f00d1b00 con 0x7f23d8004d40 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f23d800ba10 con 0x7f23f00715e0 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23d8003f30 0x7f23d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23ef7fe640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 0x7f23d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.057+0000 7f23effff640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8004d40 0x7f23d8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.058+0000 7f23eeffd640 1 -- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 msgr2=0x7f23d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.058+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f00715e0 0x7f23d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.058+0000 7f23eeffd640 1 -- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8004d40 msgr2=0x7f23d8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.058+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8004d40 0x7f23d8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.058+0000 7f23eeffd640 1 -- 172.21.15.146:0/1505082493 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d8003520 con 0x7f23d8003f30 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.058+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23d8003f30 0x7f23d8004800 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009a90 tx=0x7f23d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed538baa5ad9c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4036070 con 0x7f23d8003f30 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e4031400 con 0x7f23d8003f30 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23ee7fc640 1 -- 172.21.15.146:0/1505082493 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4009690 con 0x7f23d8003f30 2024-04-25T15:22:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23f5d46640 1 -- 172.21.15.146:0/1505082493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23d8003f30 msgr2=0x7f23d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1505082493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23d8003f30 0x7f23d8004800 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009a90 tx=0x7f23d8005830 comp rx=0 tx=0).stop 2024-04-25T15:22:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23effff640 1 -- 172.21.15.146:0/1505082493 reap_dead start 2024-04-25T15:22:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23f5d46640 1 -- 172.21.15.146:0/1505082493 shutdown_connections 2024-04-25T15:22:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23f5d46640 1 -- 172.21.15.146:0/1505082493 >> 172.21.15.146:0/1505082493 conn(0x7f23f009b2b0 msgr2=0x7f23f00b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23f5d46640 1 -- 172.21.15.146:0/1505082493 shutdown_connections 2024-04-25T15:22:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.059+0000 7f23f5d46640 1 -- 172.21.15.146:0/1505082493 wait complete. 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.061+0000 7f23f5d46640 1 Processor -- start 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.061+0000 7f23f5d46640 1 -- start start 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.062+0000 7f23f5d46640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8003f30 0x7f23f01634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.062+0000 7f23f5d46640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f009f680 0x7f23f0162a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.062+0000 7f23f5d46640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 0x7f23f009d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.062+0000 7f23f5d46640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23f00cf4b0 con 0x7f23f009d570 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23f5d46640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23f00d23a0 con 0x7f23d8003f30 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23f5d46640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f23f0070980 con 0x7f23f009f680 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 0x7f23f009d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 0x7f23f009d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50946/0 (socket says 172.21.15.146:50946) 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 -- 172.21.15.146:0/1640960093 learned_addr learned my addr 172.21.15.146:0/1640960093 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23ef7fe640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8003f30 0x7f23f01634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f009f680 0x7f23f0162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 -- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f009f680 msgr2=0x7f23f0162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f009f680 0x7f23f0162a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 -- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8003f30 msgr2=0x7f23f01634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8003f30 0x7f23f01634a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 -- 172.21.15.146:0/1640960093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f0161530 con 0x7f23f009d570 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23eeffd640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f009f680 0x7f23f0162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23effff640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 0x7f23f009d980 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f23f00cf8a0 tx=0x7f23e800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e800fe10 con 0x7f23f009d570 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e801d070 con 0x7f23f009d570 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f01619a0 con 0x7f23f009d570 2024-04-25T15:22:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8015670 con 0x7f23f009d570 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.063+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f0161f60 con 0x7f23f009d570 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.064+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f0161700 con 0x7f23f009d570 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.065+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f23e8015810 con 0x7f23f009d570 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.066+0000 7f23d7fff640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f23dc0745d0 0x7f23dc076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.066+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f23e8096030 con 0x7f23f009d570 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.066+0000 7f23ef7fe640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f23dc0745d0 0x7f23dc076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.066+0000 7f23ef7fe640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f23dc0745d0 0x7f23dc076ae0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f23f00745d0 tx=0x7f23e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:40.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.070+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23f0161700 con 0x7f23f009d570 2024-04-25T15:22:40.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.286+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f23f006a0e0 con 0x7f23f009d570 2024-04-25T15:22:40.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.287+0000 7f23d7fff640 1 -- 172.21.15.146:0/1640960093 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7f23e8010040 con 0x7f23f009d570 2024-04-25T15:22:40.289 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f23dc0745d0 msgr2=0x7f23dc076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f23dc0745d0 0x7f23dc076ae0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f23f00745d0 tx=0x7f23e0009040 comp rx=0 tx=0).stop 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 msgr2=0x7f23f009d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 0x7f23f009d980 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f23f00cf8a0 tx=0x7f23e800f580 comp rx=0 tx=0).stop 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 shutdown_connections 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f23dc0745d0 0x7f23dc076ae0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f009d570 0x7f23f009d980 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23f009f680 0x7f23f0162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 --2- 172.21.15.146:0/1640960093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23d8003f30 0x7f23f01634a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:40.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 >> 172.21.15.146:0/1640960093 conn(0x7f23f009b2b0 msgr2=0x7f23f00b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:40.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 shutdown_connections 2024-04-25T15:22:40.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:40.289+0000 7f23f5d46640 1 -- 172.21.15.146:0/1640960093 wait complete. 2024-04-25T15:22:40.301 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:22:40.302 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:22:40.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:40.302 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:22:40.321 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:22:40.321 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:22:40.321 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:22:40.321 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:22:40.322 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42574sec preferred_lft 42574sec 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:40.323 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::7cc7:d2ff:fef5:71b6/64 scope link 2024-04-25T15:22:40.324 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:40.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:40.324 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:22:40.389 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:22:40.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:40.391 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:22:40.449 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:22:40.449 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:22:40.449 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:22:40.449 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:22:40.449 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:22:40.450 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:22:40.450 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:22:40.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:22:40.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:22:40.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:22:40.450 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:22:40.541 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:22:40.572 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:40.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:40.579 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:22:40.594 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:22:40.594 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:22:40.594 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:22:40.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:40.595 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-04-25T15:22:40.659 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-25T15:22:40.659 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:40.659 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:40.714 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:22:40.714 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:40.714 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:40.768 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:22:40.832 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:40.887 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:40.888 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:22:40.889 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:40.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:40.890 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:22:40.942 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:22:40.943 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb) 2024-04-25T15:22:40.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:22:41.008 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:22:41.066 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:41.067 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:41.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:41.069 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:22:41.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:2024-04-25T15:22:41.074+0000 7efd2220b480 -1 init, newargv = 0x7efd04005a30 newargc=15 2024-04-25T15:22:41.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[44340]: starting ceph client 2024-04-25T15:22:41.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[44340]: starting fuse 2024-04-25T15:22:41.089 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:22:41.089 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T15:22:41.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:22:41.090 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.1")) 2024-04-25T15:22:41.091 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:22:41.207 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.1.44340.asok 2024-04-25T15:22:41.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:41.217 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44340.asok 2024-04-25T15:22:41.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:41.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44340.asok status 2024-04-25T15:22:41.358 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:22:41.358 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:22:41.358 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:22:41.358 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:22:41.358 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "1", 2024-04-25T15:22:41.358 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "44340", 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb" 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 6256, 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 6256 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:22:41.359 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 1388820808 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 1388820808 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.6256 192.168.0.1:0/1388820808", 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.1:0/1388820808", 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 5, 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 30, 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 56, 2024-04-25T15:22:41.360 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:22:41.361 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:22:41.361 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:22:41.361 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:22:41.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:22:41.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.486+0000 7fbbafd84640 1 Processor -- start 2024-04-25T15:22:41.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.487+0000 7fbbafd84640 1 -- start start 2024-04-25T15:22:41.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbafd84640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbba80715e0 0x7fbba806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:41.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbafd84640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbba8070190 0x7fbba8169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:41.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbafd84640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba80705a0 con 0x7fbba80b7350 2024-04-25T15:22:41.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbafd84640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbba8071bc0 con 0x7fbba80715e0 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbafd84640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbba80d1b00 con 0x7fbba8070190 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbad31f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbba80715e0 0x7fbba806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbadb20640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbba80b7350 0x7fbba8160000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45526/0 (socket says 172.21.15.146:45526) 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbadb20640 1 -- 172.21.15.146:0/2474749617 learned_addr learned my addr 172.21.15.146:0/2474749617 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbae321640 1 --2- 172.21.15.146:0/2474749617 >> v2:172.21.15.177:3300/0 conn(0x7fbba8070190 0x7fbba8169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbad31f640 1 -- 172.21.15.146:0/2474749617 >> v2:172.21.15.177:3300/0 conn(0x7fbba8070190 msgr2=0x7fbba8169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbad31f640 1 --2- 172.21.15.146:0/2474749617 >> v2:172.21.15.177:3300/0 conn(0x7fbba8070190 0x7fbba8169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbad31f640 1 -- 172.21.15.146:0/2474749617 >> v1:172.21.15.146:6789/0 conn(0x7fbba80b7350 legacy=0x7fbba8160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbad31f640 1 -- 172.21.15.146:0/2474749617 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba80d1de0 con 0x7fbba80715e0 2024-04-25T15:22:41.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.488+0000 7fbbacb1e640 1 -- 172.21.15.146:0/2474749617 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2962732958 0 0) 0x7fbba4001490 con 0x7fbba80b7350 2024-04-25T15:22:41.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.489+0000 7fbbad31f640 1 --2- 172.21.15.146:0/2474749617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80715e0 0x7fbba806fc50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fbba80cf880 tx=0x7fbb9802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707c4ca31f274bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.489+0000 7fbbacb1e640 1 -- 172.21.15.146:0/2474749617 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98037070 con 0x7fbba80715e0 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.489+0000 7fbbacb1e640 1 -- 172.21.15.146:0/2474749617 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb98032440 con 0x7fbba80715e0 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.489+0000 7fbbacb1e640 1 -- 172.21.15.146:0/2474749617 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9802ae60 con 0x7fbba80715e0 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 -- 172.21.15.146:0/2474749617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80715e0 msgr2=0x7fbba806fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 --2- 172.21.15.146:0/2474749617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80715e0 0x7fbba806fc50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fbba80cf880 tx=0x7fbb9802a110 comp rx=0 tx=0).stop 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 -- 172.21.15.146:0/2474749617 shutdown_connections 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 --2- 172.21.15.146:0/2474749617 >> v2:172.21.15.177:3300/0 conn(0x7fbba8070190 0x7fbba8169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 --2- 172.21.15.146:0/2474749617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80715e0 0x7fbba806fc50 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 -- 172.21.15.146:0/2474749617 >> 172.21.15.146:0/2474749617 conn(0x7fbba809b2b0 msgr2=0x7fbba80b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 -- 172.21.15.146:0/2474749617 shutdown_connections 2024-04-25T15:22:41.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.490+0000 7fbbafd84640 1 -- 172.21.15.146:0/2474749617 wait complete. 2024-04-25T15:22:41.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.492+0000 7fbbafd84640 1 Processor -- start 2024-04-25T15:22:41.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.492+0000 7fbbafd84640 1 -- start start 2024-04-25T15:22:41.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbafd84640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbba8070190 0x7fbba815ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:41.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbafd84640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbba80715e0 0x7fbba81e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:41.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbafd84640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 0x7fbba81e7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbafd84640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80cf4b0 con 0x7fbba80b7350 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbafd84640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80d23a0 con 0x7fbba8070190 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbafd84640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbba8070980 con 0x7fbba80715e0 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbae321640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 0x7fbba81e7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbadb20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbba8070190 0x7fbba815ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbad31f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbba80715e0 0x7fbba81e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbae321640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 0x7fbba81e7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49876/0 (socket says 172.21.15.146:49876) 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.493+0000 7fbbae321640 1 -- 172.21.15.146:0/3587112395 learned_addr learned my addr 172.21.15.146:0/3587112395 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbae321640 1 -- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbba80715e0 msgr2=0x7fbba81e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbae321640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbba80715e0 0x7fbba81e70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbae321640 1 -- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbba8070190 msgr2=0x7fbba815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbae321640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbba8070190 0x7fbba815ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbae321640 1 -- 172.21.15.146:0/3587112395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba81ea4a0 con 0x7fbba80b7350 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbad31f640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbba80715e0 0x7fbba81e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbae321640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 0x7fbba81e7600 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fbba000e820 tx=0x7fbba000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba0017070 con 0x7fbba80b7350 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba000ee70 con 0x7fbba80b7350 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba000cb60 con 0x7fbba80b7350 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba81ea780 con 0x7fbba80b7350 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.494+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba81ead40 con 0x7fbba80b7350 2024-04-25T15:22:41.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.495+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba806a0e0 con 0x7fbba80b7350 2024-04-25T15:22:41.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.496+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fbba001a070 con 0x7fbba80b7350 2024-04-25T15:22:41.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.496+0000 7fbb9e7fc640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb8c074600 0x7fbb8c076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:41.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.496+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fbba0095620 con 0x7fbba80b7350 2024-04-25T15:22:41.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.496+0000 7fbbadb20640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb8c074600 0x7fbb8c076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:41.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.497+0000 7fbbadb20640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb8c074600 0x7fbb8c076b10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbba40012e0 tx=0x7fbba4001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:41.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.501+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbba005f520 con 0x7fbba80b7350 2024-04-25T15:22:41.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.715+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbba80b6f60 con 0x7fbba80b7350 2024-04-25T15:22:41.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.715+0000 7fbb9e7fc640 1 -- 172.21.15.146:0/3587112395 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7fbba0063820 con 0x7fbba80b7350 2024-04-25T15:22:41.717 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:22:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb8c074600 msgr2=0x7fbb8c076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb8c074600 0x7fbb8c076b10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbba40012e0 tx=0x7fbba4001630 comp rx=0 tx=0).stop 2024-04-25T15:22:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 msgr2=0x7fbba81e7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 0x7fbba81e7600 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fbba000e820 tx=0x7fbba000ec20 comp rx=0 tx=0).stop 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 shutdown_connections 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb8c074600 0x7fbb8c076b10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba80b7350 0x7fbba81e7600 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbba80715e0 0x7fbba81e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 --2- 172.21.15.146:0/3587112395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbba8070190 0x7fbba815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 >> 172.21.15.146:0/3587112395 conn(0x7fbba809b2b0 msgr2=0x7fbba806df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 shutdown_connections 2024-04-25T15:22:41.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41.717+0000 7fbbafd84640 1 -- 172.21.15.146:0/3587112395 wait complete. 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb 2024-04-25T15:22:41.730 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:22:41.731 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:22:41.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:41.731 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42599sec preferred_lft 42599sec 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::e847:d767:e31b:b9f8/64 scope link noprefixroute 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::20d4:10ff:fede:a5e9/64 scope link 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:f2:8c:8f:5f:2a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::90f2:8cff:fe8f:5f2a/64 scope link 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 7a:20:01:b4:c7:f1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7820:1ff:feb4:c7f1/64 scope link 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:41.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:41.749 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:22:41.809 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-25T15:22:41.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:41.810 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-25T15:22:41.866 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-25T15:22:41.866 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:22:41.866 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:22:41.866 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:22:41.866 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:22:41.866 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:22:41.866 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-25T15:22:41.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:22:41.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:22:41.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:22:41.867 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:22:41.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:22:41.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:41.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:41.990 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:22:42.005 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:22:42.005 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:22:42.005 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:22:42.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:42.005 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.1 up 2024-04-25T15:22:42.069 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-25T15:22:42.070 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:42.070 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T15:22:42.124 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:22:42.124 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:42.125 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T15:22:42.179 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:22:42.243 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:42.298 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:22:42.299 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:42.301 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:42.302 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:42.302 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:42.302 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:42.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:42.303 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:22:42.358 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:22:42.358 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-25T15:22:42.358 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb) 2024-04-25T15:22:42.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-04-25T15:22:42.424 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:42.482 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-04-25T15:22:42.482 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-04-25T15:22:42.482 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-04-25T15:22:42.482 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-04-25T15:22:42.482 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-04-25T15:22:42.482 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-04-25T15:22:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-04-25T15:22:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:42.485 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:22:42.485 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:42.485 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:22:42.485 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:42.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:42.485 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-04-25T15:22:42.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:2024-04-25T15:22:42.489+0000 7fabb43c8480 -1 init, newargv = 0x55fd7f1fb390 newargc=15 2024-04-25T15:22:42.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[59786]: starting ceph client 2024-04-25T15:22:42.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[59786]: starting fuse 2024-04-25T15:22:42.505 INFO:teuthology.orchestra.run.smithi146.stdout:43 2024-04-25T15:22:42.506 INFO:teuthology.orchestra.run.smithi146.stdout:44 2024-04-25T15:22:42.506 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-25T15:22:42.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:22:42.506 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-04-25T15:22:42.506 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:42.507 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-04-25T15:22:42.508 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-04-25T15:22:42.509 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-04-25T15:22:42.509 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:22:42.509 DEBUG:teuthology.orchestra.run.smithi146:> 2024-04-25T15:22:42.509 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.2")) 2024-04-25T15:22:42.509 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-04-25T15:22:42.622 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.2.59786.asok 2024-04-25T15:22:42.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:42.632 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.59786.asok 2024-04-25T15:22:42.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:42.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.59786.asok status 2024-04-25T15:22:42.773 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "2", 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "59786", 2024-04-25T15:22:42.774 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb" 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6777, 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 6777 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3578801801 2024-04-25T15:22:42.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3578801801 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.6777 192.168.0.2:0/3578801801", 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.2:0/3578801801", 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 5, 2024-04-25T15:22:42.776 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 30, 2024-04-25T15:22:42.777 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 56, 2024-04-25T15:22:42.777 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:22:42.777 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-04-25T15:22:42.777 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-04-25T15:22:42.777 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:42.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:22:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.888+0000 7f4fda213640 1 Processor -- start 2024-04-25T15:22:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.888+0000 7f4fda213640 1 -- start start 2024-04-25T15:22:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fda213640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4fd4153ca0 0x7f4fd415dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fda213640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd40cf420 con 0x7f4fd40cbc90 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fda213640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd40cc270 con 0x7f4fd40d0ac0 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fda213640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f4fd40cfd10 con 0x7f4fd4153ca0 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4fd40cbc90 0x7f4fd40d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45536/0 (socket says 172.21.15.146:45536) 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd37fe640 1 -- 172.21.15.146:0/2474811286 learned_addr learned my addr 172.21.15.146:0/2474811286 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd2ffd640 1 --2- 172.21.15.146:0/2474811286 >> v2:172.21.15.177:3300/0 conn(0x7f4fd4153ca0 0x7f4fd415dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416614098 0 0) 0x7f4fd40cf420 con 0x7f4fd40cbc90 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fbc003520 con 0x7f4fd40cbc90 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2902627104 0 0) 0x7f4fbc003520 con 0x7f4fd40cbc90 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 >> v2:172.21.15.177:3300/0 conn(0x7f4fd4153ca0 msgr2=0x7f4fd415dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 --2- 172.21.15.146:0/2474811286 >> v2:172.21.15.177:3300/0 conn(0x7f4fd4153ca0 0x7f4fd415dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 >> v1:172.21.15.177:6789/0 conn(0x7f4fd40d0ac0 legacy=0x7f4fd40d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd40cf6c0 con 0x7f4fd40cbc90 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f4fc4001670 con 0x7f4fd40cbc90 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 >> v1:172.21.15.146:6789/0 conn(0x7f4fd40cbc90 legacy=0x7f4fd40d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fbc0039b0 0x7f4fbc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fbc004800 0x7f4fbc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd40cfd10 con 0x7f4fbc0051c0 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4fbc00c160 con 0x7f4fbc0039b0 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4fbc00c360 con 0x7f4fbc004800 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.890+0000 7f4fd2ffd640 1 --2- 172.21.15.146:0/2474811286 >> v2:172.21.15.177:3300/0 conn(0x7f4fd4153ca0 0x7f4fd415dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd37fe640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fbc0039b0 0x7f4fbc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd2ffd640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fbc004800 0x7f4fbc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 -- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fbc004800 msgr2=0x7f4fbc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:42.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fbc004800 0x7f4fbc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 -- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fbc0039b0 msgr2=0x7f4fbc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fbc0039b0 0x7f4fbc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 -- 172.21.15.146:0/2474811286 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fbc003520 con 0x7f4fbc0051c0 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 -- 172.21.15.146:0/2474811286 reap_dead start 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.891+0000 7f4fd3fff640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fbc00b9c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc001e20 tx=0x7f4fbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedad4a50615add6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc001a70 con 0x7f4fbc0051c0 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fcc02ad70 con 0x7f4fbc0051c0 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fd27fc640 1 -- 172.21.15.146:0/2474811286 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc003bb0 con 0x7f4fbc0051c0 2024-04-25T15:22:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 -- 172.21.15.146:0/2474811286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 msgr2=0x7f4fbc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fbc00b9c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc001e20 tx=0x7f4fbc005830 comp rx=0 tx=0).stop 2024-04-25T15:22:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 -- 172.21.15.146:0/2474811286 shutdown_connections 2024-04-25T15:22:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 --2- 172.21.15.146:0/2474811286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fbc00b9c0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 -- 172.21.15.146:0/2474811286 >> 172.21.15.146:0/2474811286 conn(0x7f4fd40ce0c0 msgr2=0x7f4fd40ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 -- 172.21.15.146:0/2474811286 shutdown_connections 2024-04-25T15:22:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.892+0000 7f4fda213640 1 -- 172.21.15.146:0/2474811286 wait complete. 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.894+0000 7f4fda213640 1 Processor -- start 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.894+0000 7f4fda213640 1 -- start start 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fd414fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fd40b6bc0 0x7f4fd4070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd40b6fd0 0x7f4fd40706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd40d01a0 con 0x7f4fbc0051c0 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41540b0 con 0x7f4fd40b6fd0 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd40d0320 con 0x7f4fd40b6bc0 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fd414fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fd414fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49886/0 (socket says 172.21.15.146:49886) 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 -- 172.21.15.146:0/1853532997 learned_addr learned my addr 172.21.15.146:0/1853532997 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd2ffd640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fd40b6bc0 0x7f4fd4070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd3fff640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd40b6fd0 0x7f4fd40706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 -- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fd40b6bc0 msgr2=0x7f4fd4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fd40b6bc0 0x7f4fd4070170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 -- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd40b6fd0 msgr2=0x7f4fd40706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd40b6fd0 0x7f4fd40706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 -- 172.21.15.146:0/1853532997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd4150340 con 0x7f4fbc0051c0 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd3fff640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd40b6fd0 0x7f4fd40706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fd37fe640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fd414fc60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4007340 tx=0x7f4fc4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc400f8c0 con 0x7f4fbc0051c0 2024-04-25T15:22:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc4018070 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd4150870 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc400fca0 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.896+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd406d2d0 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.898+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd406cde0 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.898+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4fc401d460 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.899+0000 7f4fbbfff640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4fc00745d0 0x7f4fc0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.899+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f4fc4095480 con 0x7f4fbc0051c0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.899+0000 7f4fd2ffd640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4fc00745d0 0x7f4fc0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.899+0000 7f4fd2ffd640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4fc00745d0 0x7f4fc0076ae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4070d10 tx=0x7f4fc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:42.904+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fc405fb20 con 0x7f4fbc0051c0 2024-04-25T15:22:43.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.117+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4fd406a0e0 con 0x7f4fbc0051c0 2024-04-25T15:22:43.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.118+0000 7f4fbbfff640 1 -- 172.21.15.146:0/1853532997 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7f4fc405f420 con 0x7f4fbc0051c0 2024-04-25T15:22:43.120 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4fc00745d0 msgr2=0x7f4fc0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4fc00745d0 0x7f4fc0076ae0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4070d10 tx=0x7f4fc8008040 comp rx=0 tx=0).stop 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 msgr2=0x7f4fd414fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fd414fc60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4007340 tx=0x7f4fc4007800 comp rx=0 tx=0).stop 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 shutdown_connections 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4fc00745d0 0x7f4fc0076ae0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd40b6fd0 0x7f4fd40706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fd40b6bc0 0x7f4fd4070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 --2- 172.21.15.146:0/1853532997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4fbc0051c0 0x7f4fd414fc60 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:43.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 >> 172.21.15.146:0/1853532997 conn(0x7f4fd40ce0c0 msgr2=0x7f4fd415f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:43.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 shutdown_connections 2024-04-25T15:22:43.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:43.120+0000 7f4fda213640 1 -- 172.21.15.146:0/1853532997 wait complete. 2024-04-25T15:22:43.133 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:22:43.133 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:22:43.133 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-04-25T15:22:43.134 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T15:22:43.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T15:22:43.134 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:22:43.134 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb 2024-04-25T15:22:43.134 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:22:43.134 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:22:43.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.135 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-04-25T15:22:43.154 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42571sec preferred_lft 42571sec 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::6e2f:875d:9b37:be6e/64 scope link noprefixroute 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.155 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::cba:e4ff:fef0:f38f/64 scope link 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether aa:cf:5d:6f:db:68 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::a8cf:5dff:fe6f:db68/64 scope link 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 7e:c7:d2:f5:71:b6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::7cc7:d2ff:fef5:71b6/64 scope link 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:22:43.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.156 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:22:43.220 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-04-25T15:22:43.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.223 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-04-25T15:22:43.280 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-04-25T15:22:43.280 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:22:43.281 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:22:43.281 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:22:43.281 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:22:43.281 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:22:43.281 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-04-25T15:22:43.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:22:43.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:22:43.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:22:43.281 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:22:43.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:22:43.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:43.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.411 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:22:43.426 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:22:43.426 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:22:43.426 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:22:43.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.426 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.1 up 2024-04-25T15:22:43.489 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-25T15:22:43.490 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:43.490 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T15:22:43.545 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:22:43.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:22:43.545 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T15:22:43.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:22:43.664 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:43.719 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:22:43.720 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:22:43.721 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:43.722 INFO:teuthology.orchestra.run.smithi177.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:43.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.723 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:22:43.774 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:22:43.774 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-25T15:22:43.775 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb) 2024-04-25T15:22:43.817 DEBUG:teuthology.orchestra.run.smithi177:> sudo modprobe fuse 2024-04-25T15:22:43.840 DEBUG:teuthology.orchestra.run.smithi177:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:22:43.898 INFO:teuthology.orchestra.run.smithi177.stdout:/proc 2024-04-25T15:22:43.898 INFO:teuthology.orchestra.run.smithi177.stdout:/sys 2024-04-25T15:22:43.898 INFO:teuthology.orchestra.run.smithi177.stdout:/dev 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/security 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/shm 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/pts 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/run 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/cgroup 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/pstore 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/bpf 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/ 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/selinux 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/mqueue 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/dev/hugepages 2024-04-25T15:22:43.899 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/debug 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/tracing 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/fs/fuse/connections 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/sys/kernel/config 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/run/user/1000 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T15:22:43.900 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run.smithi177.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T15:22:43.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:43.901 DEBUG:teuthology.orchestra.run.smithi177:> ls /sys/fs/fuse/connections 2024-04-25T15:22:43.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:2024-04-25T15:22:43.906+0000 7f060404a480 -1 init, newargv = 0x7f05e8056020 newargc=15 2024-04-25T15:22:43.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[44680]: starting ceph client 2024-04-25T15:22:43.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[44680]: starting fuse 2024-04-25T15:22:43.921 INFO:teuthology.orchestra.run.smithi177.stdout:43 2024-04-25T15:22:43.921 INFO:teuthology.orchestra.run.smithi177.stdout:44 2024-04-25T15:22:43.921 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> import glob 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> import re 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> import os 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> import subprocess 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> def _find_admin_socket(client_name): 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> files = glob.glob(asok_path) 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:43.922 DEBUG:teuthology.orchestra.run.smithi177:> # Given a non-glob path, it better be there 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> if "*" not in asok_path: 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> assert(len(files) == 1) 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> return files[0] 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> for f in files: 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> contents = proc_f.read() 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> if mountpoint in contents: 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> return f 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> print(_find_admin_socket("client.3")) 2024-04-25T15:22:43.923 DEBUG:teuthology.orchestra.run.smithi177:> ' 2024-04-25T15:22:44.038 INFO:teuthology.orchestra.run.smithi177.stdout:/var/run/ceph/ceph-client.3.44680.asok 2024-04-25T15:22:44.041 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:22:44.048 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44680.asok 2024-04-25T15:22:44.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:44.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44680.asok status 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "metadata": { 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "entity_id": "3", 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "hostname": "smithi177", 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "pid": "44680", 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: "root": "/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb" 2024-04-25T15:22:44.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_count": 0, 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "dentry_pinned_count": 0, 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "id": 5702, 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "inst": { 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "name": { 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "client", 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "num": 5702 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 806242545 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-25T15:22:44.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": { 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "v1", 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "addr": "192.168.0.2:0", 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "nonce": 806242545 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inst_str": "client.5702 192.168.0.2:0/806242545", 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "addr_str": "192.168.0.2:0/806242545", 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inode_count": 5, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_epoch": 30, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch": 56, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "blocklisted": false, 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout: "fs_name": "cephfs" 2024-04-25T15:22:44.192 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-25T15:22:44.199 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:22:44.199 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:44.199 DEBUG:teuthology.orchestra.run.smithi146:> pushd `mktemp -d -p .` 2024-04-25T15:22:44.199 DEBUG:teuthology.orchestra.run.smithi146:> cp -a /usr . 2024-04-25T15:22:44.199 DEBUG:teuthology.orchestra.run.smithi146:> popd 2024-04-25T15:22:44.199 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:22:44.202 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:22:44.202 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:44.202 DEBUG:teuthology.orchestra.run.smithi177:> pushd `mktemp -d -p .` 2024-04-25T15:22:44.202 DEBUG:teuthology.orchestra.run.smithi177:> cp -a /usr . 2024-04-25T15:22:44.203 DEBUG:teuthology.orchestra.run.smithi177:> popd 2024-04-25T15:22:44.203 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:22:44.205 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:22:44.205 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:44.205 DEBUG:teuthology.orchestra.run.smithi146:> pushd `mktemp -d -p .` 2024-04-25T15:22:44.205 DEBUG:teuthology.orchestra.run.smithi146:> cp -a /usr . 2024-04-25T15:22:44.205 DEBUG:teuthology.orchestra.run.smithi146:> popd 2024-04-25T15:22:44.205 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-25T15:22:44.208 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:22:44.208 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:22:44.208 DEBUG:teuthology.orchestra.run.smithi177:> pushd `mktemp -d -p .` 2024-04-25T15:22:44.208 DEBUG:teuthology.orchestra.run.smithi177:> cp -a /usr . 2024-04-25T15:22:44.208 DEBUG:teuthology.orchestra.run.smithi177:> popd 2024-04-25T15:22:44.208 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-04-25T15:22:44.254 INFO:teuthology.orchestra.run.smithi177.stderr:++ mktemp -d -p . 2024-04-25T15:22:44.261 INFO:teuthology.orchestra.run.smithi177.stderr:+ pushd ./tmp.v5WVmX6aR6 2024-04-25T15:22:44.261 INFO:teuthology.orchestra.run.smithi177.stderr:++ mktemp -d -p . 2024-04-25T15:22:44.262 INFO:teuthology.orchestra.run.smithi177.stdout:~/cephtest/mnt.1/tmp.v5WVmX6aR6 ~/cephtest/mnt.1 2024-04-25T15:22:44.262 INFO:teuthology.orchestra.run.smithi177.stderr:+ cp -a /usr . 2024-04-25T15:22:44.262 INFO:teuthology.orchestra.run.smithi146.stderr:++ mktemp -d -p . 2024-04-25T15:22:44.267 INFO:teuthology.orchestra.run.smithi177.stderr:+ pushd ./tmp.Q9vxbLIvBX 2024-04-25T15:22:44.275 INFO:teuthology.orchestra.run.smithi177.stdout:~/cephtest/mnt.3/tmp.Q9vxbLIvBX ~/cephtest/mnt.3 2024-04-25T15:22:44.275 INFO:teuthology.orchestra.run.smithi177.stderr:+ cp -a /usr . 2024-04-25T15:22:44.275 INFO:teuthology.orchestra.run.smithi146.stderr:+ pushd ./tmp.qYoo3Qr6Mh 2024-04-25T15:22:44.275 INFO:teuthology.orchestra.run.smithi146.stderr:++ mktemp -d -p . 2024-04-25T15:22:44.276 INFO:teuthology.orchestra.run.smithi146.stdout:~/cephtest/mnt.0/tmp.qYoo3Qr6Mh ~/cephtest/mnt.0 2024-04-25T15:22:44.276 INFO:teuthology.orchestra.run.smithi146.stderr:+ cp -a /usr . 2024-04-25T15:22:44.283 INFO:teuthology.orchestra.run.smithi146.stderr:+ pushd ./tmp.pQIyEuvXQz 2024-04-25T15:22:44.283 INFO:teuthology.orchestra.run.smithi146.stdout:~/cephtest/mnt.2/tmp.pQIyEuvXQz ~/cephtest/mnt.2 2024-04-25T15:22:44.283 INFO:teuthology.orchestra.run.smithi146.stderr:+ cp -a /usr . 2024-04-25T15:22:46.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:22:46.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:22:48.452 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/sudo' for reading: Permission denied 2024-04-25T15:22:48.456 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/sudo' for reading: Permission denied 2024-04-25T15:22:48.459 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/sudo' for reading: Permission denied 2024-04-25T15:22:48.461 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/sudo' for reading: Permission denied 2024-04-25T15:22:48.462 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/sudoreplay' for reading: Permission denied 2024-04-25T15:22:48.466 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/sudoreplay' for reading: Permission denied 2024-04-25T15:22:48.468 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/sudoreplay' for reading: Permission denied 2024-04-25T15:22:48.469 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/sudoreplay' for reading: Permission denied 2024-04-25T15:22:49.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 quiesce path /volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb 2024-04-25T15:22:49.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.342+0000 7fbcb9813640 1 Processor -- start 2024-04-25T15:22:49.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.343+0000 7fbcb9813640 1 -- start start 2024-04-25T15:22:49.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb9813640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb406e560 0x7fbcb40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb9813640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbcb40ceb70 0x7fbcb4166c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb9813640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcb40cce80 con 0x7fbcb41606b0 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb9813640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbcb40cd160 con 0x7fbcb40ceb70 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb9813640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb40cd440 con 0x7fbcb406e560 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb2ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbcb41606b0 0x7fbcb40cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45548/0 (socket says 172.21.15.146:45548) 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/3394009304 learned_addr learned my addr 172.21.15.146:0/3394009304 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:49.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb406e560 0x7fbcb40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.344+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/3394009304 >> v2:172.21.15.177:3300/0 conn(0x7fbcb40ceb70 0x7fbcb4166c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcb27fc640 1 -- 172.21.15.146:0/3394009304 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990441551 0 0) 0x7fbcb40cce80 con 0x7fbcb41606b0 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcb27fc640 1 -- 172.21.15.146:0/3394009304 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbca0003520 con 0x7fbcb41606b0 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcb37fe640 1 -- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb406e560 msgr2=0x7fbcb40ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb406e560 0x7fbcb40ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcb37fe640 1 -- 172.21.15.146:0/3394009304 >> v1:172.21.15.146:6789/0 conn(0x7fbcb41606b0 legacy=0x7fbcb40cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcb37fe640 1 -- 172.21.15.146:0/3394009304 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb40cd720 con 0x7fbcb40ceb70 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.345+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb406e560 0x7fbcb40ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.346+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb40ceb70 0x7fbcb4166c50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009d90 tx=0x7fbcac02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f647bde266eb99bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.346+0000 7fbcb27fc640 1 -- 172.21.15.146:0/3394009304 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac007740 con 0x7fbcb40ceb70 2024-04-25T15:22:49.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.346+0000 7fbcb27fc640 1 -- 172.21.15.146:0/3394009304 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcac0078e0 con 0x7fbcb40ceb70 2024-04-25T15:22:49.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.346+0000 7fbcb27fc640 1 -- 172.21.15.146:0/3394009304 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac007cb0 con 0x7fbcb40ceb70 2024-04-25T15:22:49.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 -- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb40ceb70 msgr2=0x7fbcb4166c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb40ceb70 0x7fbcb4166c50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009d90 tx=0x7fbcac02a660 comp rx=0 tx=0).stop 2024-04-25T15:22:49.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 -- 172.21.15.146:0/3394009304 shutdown_connections 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb40ceb70 0x7fbcb4166c50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 --2- 172.21.15.146:0/3394009304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb406e560 0x7fbcb40ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 -- 172.21.15.146:0/3394009304 >> 172.21.15.146:0/3394009304 conn(0x7fbcb409b2b0 msgr2=0x7fbcb40b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 -- 172.21.15.146:0/3394009304 shutdown_connections 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.347+0000 7fbcb9813640 1 -- 172.21.15.146:0/3394009304 wait complete. 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.349+0000 7fbcb9813640 1 Processor -- start 2024-04-25T15:22:49.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.349+0000 7fbcb9813640 1 -- start start 2024-04-25T15:22:49.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb9813640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb406e560 0x7fbcb41f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb9813640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb40ceb70 0x7fbcb4159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb9813640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 0x7fbcb40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb9813640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb40ccb90 con 0x7fbcb41606b0 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb9813640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb40cca10 con 0x7fbcb406e560 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb9813640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb40cef80 con 0x7fbcb40ceb70 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 0x7fbcb40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.350+0000 7fbcb37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 0x7fbcb40c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49896/0 (socket says 172.21.15.146:49896) 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 -- 172.21.15.146:0/4110087340 learned_addr learned my addr 172.21.15.146:0/4110087340 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb406e560 0x7fbcb41f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb40ceb70 0x7fbcb4159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 -- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb40ceb70 msgr2=0x7fbcb4159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb40ceb70 0x7fbcb4159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 -- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb406e560 msgr2=0x7fbcb41f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb406e560 0x7fbcb41f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 -- 172.21.15.146:0/4110087340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb40ca8a0 con 0x7fbcb41606b0 2024-04-25T15:22:49.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb40ceb70 0x7fbcb4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:49.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 0x7fbcb40c7080 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009eb0 tx=0x7fbcac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb0ff9640 1 -- 172.21.15.146:0/4110087340 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac03f070 con 0x7fbcb41606b0 2024-04-25T15:22:49.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb0ff9640 1 -- 172.21.15.146:0/4110087340 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcac031bc0 con 0x7fbcb41606b0 2024-04-25T15:22:49.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb0ff9640 1 -- 172.21.15.146:0/4110087340 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac009450 con 0x7fbcb41606b0 2024-04-25T15:22:49.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb40ca2e0 con 0x7fbcb41606b0 2024-04-25T15:22:49.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.351+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb40c5b30 con 0x7fbcb41606b0 2024-04-25T15:22:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.354+0000 7fbcb0ff9640 1 -- 172.21.15.146:0/4110087340 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fbcac03b410 con 0x7fbcb41606b0 2024-04-25T15:22:49.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.354+0000 7fbcb0ff9640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbc94074600 0x7fbc94076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.354+0000 7fbcb0ff9640 1 -- 172.21.15.146:0/4110087340 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fbcac0b6840 con 0x7fbcb41606b0 2024-04-25T15:22:49.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.354+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbc94074600 0x7fbc94076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.355+0000 7fbcc07a0740 1 Processor -- start 2024-04-25T15:22:49.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.355+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbc94074600 0x7fbc94076b10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbca4005880 tx=0x7fbca4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.355+0000 7fbcc07a0740 1 -- start start 2024-04-25T15:22:49.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5611fe7ea150 con 0x5611fe4babd0 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5611fe785c50 con 0x5611fe4e3a90 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5611fe7e6110 con 0x5611fe4c2db0 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcaa7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8af5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47374/0 (socket says 172.21.15.146:47374) 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/1847727463 learned_addr learned my addr 172.21.15.146:0/1847727463 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 msgr2=0x5611fe8ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/1847727463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 msgr2=0x5611fe8ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/1847727463 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611fe7b2fc0 con 0x5611fe4e3a90 2024-04-25T15:22:49.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.357+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:49.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.358+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8af5c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbca400fae0 tx=0x7fbca4000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eca9bc574486f02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.358+0000 7fbca97fa640 1 -- 172.21.15.146:0/1847727463 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca4025070 con 0x5611fe4e3a90 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.358+0000 7fbca97fa640 1 -- 172.21.15.146:0/1847727463 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca4015bf0 con 0x5611fe4e3a90 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.358+0000 7fbca97fa640 1 -- 172.21.15.146:0/1847727463 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca4020400 con 0x5611fe4e3a90 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 -- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 msgr2=0x5611fe8af5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8af5c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbca400fae0 tx=0x7fbca4000ed0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 -- 172.21.15.146:0/1847727463 shutdown_connections 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/1847727463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8af5c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 -- 172.21.15.146:0/1847727463 >> 172.21.15.146:0/1847727463 conn(0x5611fe4d7be0 msgr2=0x5611fe884c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 -- 172.21.15.146:0/1847727463 shutdown_connections 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.359+0000 7fbcc07a0740 1 -- 172.21.15.146:0/1847727463 wait complete. 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.360+0000 7fbcc07a0740 1 -- start start 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.360+0000 7fbcc07a0740 1 Processor -- start 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcb2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5611fe770be0 con 0x5611fe4babd0 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcb2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8b1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49906/0 (socket says 172.21.15.146:49906) 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5611fe8daa00 con 0x5611fe4e3a90 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/3071442420 learned_addr learned my addr 172.21.15.146:0/3071442420 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5611fe8b2760 con 0x5611fe4c2db0 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.361+0000 7fbcc07a0740 10 client.0 _ll_register_callbacks cb 0x5611fe7e1710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 msgr2=0x5611fe8ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 msgr2=0x5611fe8ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/3071442420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611fe4fbaf0 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8b1c80 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fbca4008fd0 tx=0x7fbca4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60df21182e70848c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbc927fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbc927fc640 1 -- 172.21.15.146:0/3071442420 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca402f070 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbc927fc640 1 -- 172.21.15.146:0/3071442420 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fbca4020af0 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbca4015bf0 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbc927fc640 1 -- 172.21.15.146:0/3071442420 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbca4039050 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcc07a0740 10 client.6801 fetch_fsmap learned FSMap version 30 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5611fe5c7f00 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.362+0000 7fbc927fc640 1 -- 172.21.15.146:0/3071442420 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca4015bf0 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbc927fc640 1 -- 172.21.15.146:0/3071442420 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fbca403c020 con 0x5611fe4babd0 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcc07a0740 10 client.6801 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T15:22:49.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcc07a0740 10 client.6801 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcc07a0740 20 client.6801 populate_metadata read hostname 'smithi146' 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe4a2660 0x5611fe8b0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcc07a0740 4 client.6801 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 --> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611fe7f7d60 con 0x5611fe4a2660 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe4a2660 0x5611fe8b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbc927fc640 10 client.6801 ms_handle_connect on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.363+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe4a2660 0x5611fe8b0910 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e317ddd8db969fc server_cookie=b0b06eaedba64e04 in_seq=0 out_seq=0 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.364+0000 7fbc927fc640 1 -- 172.21.15.146:0/3071442420 <== mds.0 v2:172.21.15.146:6834/865508848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5611fe7f7d60 con 0x5611fe4a2660 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.364+0000 7fbc927fc640 10 client.6801 handle_command_reply: tid=0 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 client.6801 shutdown 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 msgr2=0x5611fe8b1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8b1c80 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fbca4008fd0 tx=0x7fbca4000bc0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 shutdown_connections 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe4a2660 0x5611fe8b0910 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4e3a90 0x5611fe8ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbc927fc640 0 client.6801 ms_handle_reset on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4c2db0 0x5611fe8ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3071442420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8b1c80 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 >> 172.21.15.146:0/3071442420 conn(0x5611fe4d7be0 msgr2=0x5611fe884c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 shutdown_connections 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3071442420 wait complete. 2024-04-25T15:22:49.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.365+0000 7fbcc07a0740 20 client.6801 trim_cache size 0 max 16384 2024-04-25T15:22:49.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.385+0000 7fbcc07a0740 1 Processor -- start 2024-04-25T15:22:49.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.385+0000 7fbcc07a0740 1 -- start start 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.386+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4a2660 0x5611fe8d9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.386+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.386+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe8dc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.386+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb41fecb0 con 0x5611fe4babd0 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.386+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb41fdd70 con 0x5611fe4c2db0 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.386+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb41f30f0 con 0x5611fe4a2660 2024-04-25T15:22:49.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8dbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49920/0 (socket says 172.21.15.146:49920) 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 -- 172.21.15.146:0/3253880636 learned_addr learned my addr 172.21.15.146:0/3253880636 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4a2660 0x5611fe8d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe8dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 -- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4a2660 msgr2=0x5611fe8d9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4a2660 0x5611fe8d9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 -- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 msgr2=0x5611fe8dc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe8dc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 -- 172.21.15.146:0/3253880636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611fe87f1a0 con 0x5611fe4babd0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8dbd50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4069590 tx=0x7fbc9c013d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc8ab6977b9499e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbca8ff9640 1 -- 172.21.15.146:0/3253880636 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c004230 con 0x5611fe4babd0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbca8ff9640 1 -- 172.21.15.146:0/3253880636 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c027070 con 0x5611fe4babd0 2024-04-25T15:22:49.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.387+0000 7fbca8ff9640 1 -- 172.21.15.146:0/3253880636 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c0108f0 con 0x5611fe4babd0 2024-04-25T15:22:49.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3253880636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 msgr2=0x5611fe8dbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8dbd50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4069590 tx=0x7fbc9c013d20 comp rx=0 tx=0).stop 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3253880636 shutdown_connections 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe8dc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4babd0 0x5611fe8dbd50 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/3253880636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4a2660 0x5611fe8d9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3253880636 >> 172.21.15.146:0/3253880636 conn(0x5611fe4d7be0 msgr2=0x5611fe8a9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3253880636 shutdown_connections 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.388+0000 7fbcc07a0740 1 -- 172.21.15.146:0/3253880636 wait complete. 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.389+0000 7fbcc07a0740 1 -- start start 2024-04-25T15:22:49.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.389+0000 7fbcc07a0740 1 Processor -- start 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 0x5611fe946eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4babd0 0x5611fe883f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe884980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5611fe8acf30 con 0x5611fe4a2660 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb415d510 con 0x5611fe4c2db0 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x5611fe8afd90 con 0x5611fe4babd0 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcc07a0740 10 client.0 _ll_register_callbacks cb 0x7fbca4023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 0x5611fe946eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 0x5611fe946eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49934/0 (socket says 172.21.15.146:49934) 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/701235756 learned_addr learned my addr 172.21.15.146:0/701235756 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4babd0 0x5611fe883f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb37fe640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe884980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4babd0 msgr2=0x5611fe883f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4babd0 0x5611fe883f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 msgr2=0x5611fe884980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe884980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 -- 172.21.15.146:0/701235756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611fe8d8280 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbcb2ffd640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 0x5611fe946eb0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbca402a9e0 tx=0x7fbca4007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1010e83d4bbaec69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.391+0000 7fbc927fc640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbc927fc640 1 -- 172.21.15.146:0/701235756 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca402f070 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbc927fc640 1 -- 172.21.15.146:0/701235756 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7fbca40047e0 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc9c027070 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbc927fc640 1 -- 172.21.15.146:0/701235756 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbca4025040 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 10 client.6807 fetch_fsmap learned FSMap version 30 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5611fe884490 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbc927fc640 1 -- 172.21.15.146:0/701235756 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca4015bf0 con 0x5611fe4a2660 2024-04-25T15:22:49.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbc927fc640 1 -- 172.21.15.146:0/701235756 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fbca403c040 con 0x5611fe4a2660 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 10 client.6807 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 10 client.6807 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 20 client.6807 populate_metadata read hostname 'smithi146' 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe477aa0 0x5611fe8d9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 4 client.6807 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb"}] 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 --> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] -- command(tid 0: {"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/df88dd5c-68a0-43ab-b95a-f97878e0c1eb"}) -- 0x7fbc9c027070 con 0x5611fe477aa0 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.392+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe477aa0 0x5611fe8d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.393+0000 7fbc927fc640 10 client.6807 ms_handle_connect on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:49.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.393+0000 7fbcaa7fc640 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe477aa0 0x5611fe8d9890 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4fb1eff31762921 server_cookie=9764f7946de2f543 in_seq=0 out_seq=0 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.394+0000 7fbc927fc640 1 -- 172.21.15.146:0/701235756 <== mds.0 v2:172.21.15.146:6834/865508848 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1689 (crc 0 0 0) 0x7fbc9c027070 con 0x5611fe477aa0 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.394+0000 7fbc927fc640 10 client.6807 handle_command_reply: tid=0 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.394+0000 7fbcc07a0740 1 client.6807 shutdown 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.394+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 msgr2=0x5611fe946eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.394+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 0x5611fe946eb0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbca402a9e0 tx=0x7fbca4007d10 comp rx=0 tx=0).stop 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 shutdown_connections 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:6834/865508848,v1:172.21.15.146:6836/865508848] conn(0x5611fe477aa0 0x5611fe8d9890 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5611fe4c2db0 0x5611fe884980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5611fe4babd0 0x5611fe883f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 --2- 172.21.15.146:0/701235756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5611fe4a2660 0x5611fe946eb0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 >> 172.21.15.146:0/701235756 conn(0x5611fe4d7be0 msgr2=0x5611fe8ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbc927fc640 0 client.6807 ms_handle_reset on v2:172.21.15.146:6834/865508848 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 shutdown_connections 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 1 -- 172.21.15.146:0/701235756 wait complete. 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.395+0000 7fbcc07a0740 20 client.6807 trim_cache size 0 max 16384 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "op": { 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "result": -2147483648, 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "acquired locks", 2024-04-25T15:22:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "reqid": { 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "entity": { 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "mds", 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 0 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 13 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "op_type": "internal_op", 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "internal_op": 5384, 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "op_name": "quiesce_path", 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395762+0000", 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated" 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395762+0000", 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read" 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395762+0000", 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled" 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395762+0000", 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read" 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395762+0000", 2024-04-25T15:22:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched" 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395825+0000", 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395850+0000", 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395871+0000", 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-25T15:22:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-04-25T15:22:49.395891+0000", 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "acquired locks" 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "locks": null 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "state": { 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes": 1, 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_quiesced": 0, 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_blocked": 0, 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_dropped": 0, 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "failed": [] 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-25T15:22:49.404 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:49.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbc94074600 msgr2=0x7fbc94076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbc94074600 0x7fbc94076b10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbca4005880 tx=0x7fbca4009040 comp rx=0 tx=0).stop 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 msgr2=0x7fbcb40c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 0x7fbcb40c7080 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009eb0 tx=0x7fbcac004a40 comp rx=0 tx=0).stop 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 shutdown_connections 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbc94074600 0x7fbc94076b10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcb41606b0 0x7fbcb40c7080 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbcb40ceb70 0x7fbcb4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 --2- 172.21.15.146:0/4110087340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbcb406e560 0x7fbcb41f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 >> 172.21.15.146:0/4110087340 conn(0x7fbcb409b2b0 msgr2=0x7fbcb406c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 shutdown_connections 2024-04-25T15:22:49.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.396+0000 7fbcb9813640 1 -- 172.21.15.146:0/4110087340 wait complete. 2024-04-25T15:22:49.410 DEBUG:tasks.cephfs.test_quiesce:{'op': {'result': -2147483648, 'flag_point': 'acquired locks', 'reqid': {'entity': {'type': 'mds', 'num': 0}, 'tid': 13}, 'op_type': 'internal_op', 'internal_op': 5384, 'op_name': 'quiesce_path', 'events': [{'time': '2024-04-25T15:22:49.395762+0000', 'event': 'initiated'}, {'time': '2024-04-25T15:22:49.395762+0000', 'event': 'header_read'}, {'time': '2024-04-25T15:22:49.395762+0000', 'event': 'throttled'}, {'time': '2024-04-25T15:22:49.395762+0000', 'event': 'all_read'}, {'time': '2024-04-25T15:22:49.395762+0000', 'event': 'dispatched'}, {'time': '2024-04-25T15:22:49.395825+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:22:49.395850+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:22:49.395871+0000', 'event': 'acquired locks'}, {'time': '2024-04-25T15:22:49.395891+0000', 'event': 'acquired locks'}], 'locks': None}, 'state': {'inodes': 1, 'inodes_quiesced': 0, 'inodes_blocked': 0, 'inodes_dropped': 0, 'failed': []}} 2024-04-25T15:22:49.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:22:49.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.561+0000 7f1679020640 1 Processor -- start 2024-04-25T15:22:49.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.561+0000 7f1679020640 1 -- start start 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.562+0000 7f1679020640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16740d0e60 0x7f16740d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.562+0000 7f1679020640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f167406d9f0 0x7f167415fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.562+0000 7f1679020640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16740d4a30 con 0x7f1674163aa0 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.562+0000 7f1679020640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f167406c550 con 0x7f16740d0e60 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.562+0000 7f1679020640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f16740d4d10 con 0x7f167406d9f0 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16740d0e60 0x7f16740d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16740d0e60 0x7f16740d43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49944/0 (socket says 172.21.15.146:49944) 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 -- 172.21.15.146:0/778917176 learned_addr learned my addr 172.21.15.146:0/778917176 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f167359e640 1 --2- 172.21.15.146:0/778917176 >> v2:172.21.15.177:3300/0 conn(0x7f167406d9f0 0x7f167415fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 -- 172.21.15.146:0/778917176 >> v2:172.21.15.177:3300/0 conn(0x7f167406d9f0 msgr2=0x7f167415fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 --2- 172.21.15.146:0/778917176 >> v2:172.21.15.177:3300/0 conn(0x7f167406d9f0 0x7f167415fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 -- 172.21.15.146:0/778917176 >> v1:172.21.15.177:6789/0 conn(0x7f1674163aa0 legacy=0x7f16740d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.563+0000 7f166bfff640 1 -- 172.21.15.146:0/778917176 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16740d4ff0 con 0x7f16740d0e60 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.564+0000 7f166bfff640 1 --2- 172.21.15.146:0/778917176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740d0e60 0x7f16740d43a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f167408ee60 tx=0x7f165c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818f4ff6495a4f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.564+0000 7f167259c640 1 -- 172.21.15.146:0/778917176 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c037070 con 0x7f16740d0e60 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.564+0000 7f167259c640 1 -- 172.21.15.146:0/778917176 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165c02ae10 con 0x7f16740d0e60 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.564+0000 7f167259c640 1 -- 172.21.15.146:0/778917176 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c032660 con 0x7f16740d0e60 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 -- 172.21.15.146:0/778917176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740d0e60 msgr2=0x7f16740d43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 --2- 172.21.15.146:0/778917176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740d0e60 0x7f16740d43a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f167408ee60 tx=0x7f165c02a180 comp rx=0 tx=0).stop 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 -- 172.21.15.146:0/778917176 shutdown_connections 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 --2- 172.21.15.146:0/778917176 >> v2:172.21.15.177:3300/0 conn(0x7f167406d9f0 0x7f167415fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 --2- 172.21.15.146:0/778917176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740d0e60 0x7f16740d43a0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 -- 172.21.15.146:0/778917176 >> 172.21.15.146:0/778917176 conn(0x7f16740d6280 msgr2=0x7f16740b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 -- 172.21.15.146:0/778917176 shutdown_connections 2024-04-25T15:22:49.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.565+0000 7f1679020640 1 -- 172.21.15.146:0/778917176 wait complete. 2024-04-25T15:22:49.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.567+0000 7f1679020640 1 Processor -- start 2024-04-25T15:22:49.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.567+0000 7f1679020640 1 -- start start 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f1679020640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f167406d9f0 0x7f16740b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f1679020640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1674163aa0 0x7f16740cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f1679020640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740c9e30 0x7f16740ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f1679020640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1674160420 con 0x7f16740c9e30 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f1679020640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f16740d1580 con 0x7f167406d9f0 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f1679020640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f16741605a0 con 0x7f1674163aa0 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.569+0000 7f167359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740c9e30 0x7f16740ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f167406d9f0 0x7f16740b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f167359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740c9e30 0x7f16740ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49952/0 (socket says 172.21.15.146:49952) 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f167359e640 1 -- 172.21.15.146:0/431844947 learned_addr learned my addr 172.21.15.146:0/431844947 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f166bfff640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1674163aa0 0x7f16740cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 -- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1674163aa0 msgr2=0x7f16740cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1674163aa0 0x7f16740cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 -- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740c9e30 msgr2=0x7f16740ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740c9e30 0x7f16740ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 -- 172.21.15.146:0/431844947 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16740ca780 con 0x7f167406d9f0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1672d9d640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f167406d9f0 0x7f16740b4dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f166400c460 tx=0x7f166400c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664017070 con 0x7f167406d9f0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16640035f0 con 0x7f167406d9f0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16640039e0 con 0x7f167406d9f0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16740cc2e0 con 0x7f167406d9f0 2024-04-25T15:22:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.570+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16740cce60 con 0x7f167406d9f0 2024-04-25T15:22:49.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.572+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167406a0e0 con 0x7f167406d9f0 2024-04-25T15:22:49.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.572+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f166401a070 con 0x7f167406d9f0 2024-04-25T15:22:49.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.573+0000 7f166b7fe640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1654074600 0x7f1654076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.573+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6159+0+0 (secure 0 0 0) 0x7f1664094840 con 0x7f167406d9f0 2024-04-25T15:22:49.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.573+0000 7f166bfff640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1654074600 0x7f1654076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.573+0000 7f166bfff640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1654074600 0x7f1654076b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f165c034000 tx=0x7f165c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.578+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16740ca780 con 0x7f167406d9f0 2024-04-25T15:22:49.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.828+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f16740d3ec0 con 0x7f167406d9f0 2024-04-25T15:22:49.833 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-04-25T15:22:49.830+0000 7f79638be640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:22:49.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.837+0000 7f166b7fe640 1 -- 172.21.15.146:0/431844947 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) ==== 161+0+0 (secure 0 0 0) 0x7f166405e770 con 0x7f167406d9f0 2024-04-25T15:22:49.841 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.840+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1654074600 msgr2=0x7f1654076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.840+0000 7f1679020640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1654074600 0x7f1654076b10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f165c034000 tx=0x7f165c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.840+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f167406d9f0 msgr2=0x7f16740b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.840+0000 7f1679020640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f167406d9f0 0x7f16740b4dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f166400c460 tx=0x7f166400c920 comp rx=0 tx=0).stop 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.840+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 shutdown_connections 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.840+0000 7f1679020640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1654074600 0x7f1654076b10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.841+0000 7f1679020640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16740c9e30 0x7f16740ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.841+0000 7f1679020640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1674163aa0 0x7f16740cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.841+0000 7f1679020640 1 --2- 172.21.15.146:0/431844947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f167406d9f0 0x7f16740b4dc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.841+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 >> 172.21.15.146:0/431844947 conn(0x7f16740d6280 msgr2=0x7f16740b9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.841+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 shutdown_connections 2024-04-25T15:22:49.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.841+0000 7f1679020640 1 -- 172.21.15.146:0/431844947 wait complete. 2024-04-25T15:22:49.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-25T15:22:49.959 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-04-25T15:22:49.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.975+0000 7f22fedd7640 1 Processor -- start 2024-04-25T15:22:49.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.975+0000 7f22fedd7640 1 -- start start 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fedd7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f8163c00 0x7f22f806d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fedd7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f22f80bb450 0x7f22f80d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fedd7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22f80d0de0 0x7f22f80b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fedd7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f22f806dea0 con 0x7f22f80d0de0 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fedd7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f22f80b5230 con 0x7f22f80bb450 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fedd7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f22f806c8c0 con 0x7f22f8163c00 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fd374640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22f80d0de0 0x7f22f80b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fd374640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22f80d0de0 0x7f22f80b8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49972/0 (socket says 172.21.15.146:49972) 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fd374640 1 -- 172.21.15.146:0/3799700259 learned_addr learned my addr 172.21.15.146:0/3799700259 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22f7fff640 1 --2- 172.21.15.146:0/3799700259 >> v2:172.21.15.177:3300/0 conn(0x7f22f80bb450 0x7f22f80d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.976+0000 7f22fcb73640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f8163c00 0x7f22f806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fd374640 1 -- 172.21.15.146:0/3799700259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f8163c00 msgr2=0x7f22f806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fd374640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f8163c00 0x7f22f806d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fd374640 1 -- 172.21.15.146:0/3799700259 >> v2:172.21.15.177:3300/0 conn(0x7f22f80bb450 msgr2=0x7f22f80d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:49.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fd374640 1 --2- 172.21.15.146:0/3799700259 >> v2:172.21.15.177:3300/0 conn(0x7f22f80bb450 0x7f22f80d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fd374640 1 -- 172.21.15.146:0/3799700259 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f806cba0 con 0x7f22f80d0de0 2024-04-25T15:22:49.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fcb73640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f8163c00 0x7f22f806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:22:49.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22f7fff640 1 --2- 172.21.15.146:0/3799700259 >> v2:172.21.15.177:3300/0 conn(0x7f22f80bb450 0x7f22f80d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22fd374640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80d0de0 0x7f22f80b8fb0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f22f808de60 tx=0x7f22f002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c38db7b40c20ffd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22f77fe640 1 -- 172.21.15.146:0/3799700259 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0036070 con 0x7f22f80d0de0 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22f77fe640 1 -- 172.21.15.146:0/3799700259 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f0031600 con 0x7f22f80d0de0 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.977+0000 7f22f77fe640 1 -- 172.21.15.146:0/3799700259 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f00319d0 con 0x7f22f80d0de0 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 -- 172.21.15.146:0/3799700259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80d0de0 msgr2=0x7f22f80b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80d0de0 0x7f22f80b8fb0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f22f808de60 tx=0x7f22f002a610 comp rx=0 tx=0).stop 2024-04-25T15:22:49.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 -- 172.21.15.146:0/3799700259 shutdown_connections 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80d0de0 0x7f22f80b8fb0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 --2- 172.21.15.146:0/3799700259 >> v2:172.21.15.177:3300/0 conn(0x7f22f80bb450 0x7f22f80d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 --2- 172.21.15.146:0/3799700259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f8163c00 0x7f22f806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 -- 172.21.15.146:0/3799700259 >> 172.21.15.146:0/3799700259 conn(0x7f22f80b7970 msgr2=0x7f22f80b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 -- 172.21.15.146:0/3799700259 shutdown_connections 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.978+0000 7f22fedd7640 1 -- 172.21.15.146:0/3799700259 wait complete. 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.980+0000 7f22fedd7640 1 Processor -- start 2024-04-25T15:22:49.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.980+0000 7f22fedd7640 1 -- start start 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fedd7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 0x7f22f8145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fedd7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f80d0de0 0x7f22f815e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fedd7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f8163c00 0x7f22f8157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fedd7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f22f806c550 con 0x7f22f80bb450 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fedd7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22f806d160 con 0x7f22f8163c00 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fedd7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f22f80bb860 con 0x7f22f80d0de0 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fcb73640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 0x7f22f8145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fcb73640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 0x7f22f8145ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49984/0 (socket says 172.21.15.146:49984) 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fcb73640 1 -- 172.21.15.146:0/260477461 learned_addr learned my addr 172.21.15.146:0/260477461 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22f7fff640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f80d0de0 0x7f22f815e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.981+0000 7f22fd374640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f8163c00 0x7f22f8157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fcb73640 1 -- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f80d0de0 msgr2=0x7f22f815e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fcb73640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f80d0de0 0x7f22f815e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fcb73640 1 -- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f8163c00 msgr2=0x7f22f8157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fcb73640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f8163c00 0x7f22f8157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fcb73640 1 -- 172.21.15.146:0/260477461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f815f420 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fcb73640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 0x7f22f8145ce0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f22e000c7f0 tx=0x7f22e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0015070 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e0007820 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0007c10 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f815f700 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.982+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f815fcc0 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.983+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f806a0e0 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.984+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f22e0018070 con 0x7f22f80bb450 2024-04-25T15:22:49.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.984+0000 7f22f57fa640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f22d8074600 0x7f22d8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:49.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.984+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f22e0095490 con 0x7f22f80bb450 2024-04-25T15:22:49.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.984+0000 7f22f7fff640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f22d8074600 0x7f22d8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:49.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.985+0000 7f22f7fff640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f22d8074600 0x7f22d8076b10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f22f815b140 tx=0x7f22e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:49.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:49.989+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f22e005f0f0 con 0x7f22f80bb450 2024-04-25T15:22:50.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.211+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f22f80d38d0 con 0x7f22f80bb450 2024-04-25T15:22:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.841+0000 7f22f57fa640 1 -- 172.21.15.146:0/260477461 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v32) ==== 162+0+0 (secure 0 0 0) 0x7f22e00633f0 con 0x7f22f80bb450 2024-04-25T15:22:50.843 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f22d8074600 msgr2=0x7f22d8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f22d8074600 0x7f22d8076b10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f22f815b140 tx=0x7f22e8008040 comp rx=0 tx=0).stop 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 msgr2=0x7f22f8145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 0x7f22f8145ce0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f22e000c7f0 tx=0x7f22e000ccb0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 shutdown_connections 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f22d8074600 0x7f22d8076b10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f8163c00 0x7f22f8157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22f80d0de0 0x7f22f815e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 --2- 172.21.15.146:0/260477461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22f80bb450 0x7f22f8145ce0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 >> 172.21.15.146:0/260477461 conn(0x7f22f80b7970 msgr2=0x7f22f81621e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:50.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 shutdown_connections 2024-04-25T15:22:50.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.844+0000 7f22fedd7640 1 -- 172.21.15.146:0/260477461 wait complete. 2024-04-25T15:22:50.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:50.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.986+0000 7f204feaa640 1 Processor -- start 2024-04-25T15:22:50.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.986+0000 7f204feaa640 1 -- start start 2024-04-25T15:22:50.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.987+0000 7f204feaa640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20480d4f60 0x7f20480d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.987+0000 7f204feaa640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f20480d0020 0x7f20480d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.987+0000 7f204feaa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20480b9200 con 0x7f2048163d80 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.987+0000 7f204feaa640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f204806c550 con 0x7f20480d4f60 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.987+0000 7f204feaa640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f204806c830 con 0x7f20480d0020 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20480d4f60 0x7f20480d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20480d4f60 0x7f20480d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58354/0 (socket says 172.21.15.146:58354) 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 -- 172.21.15.146:0/3728038131 learned_addr learned my addr 172.21.15.146:0/3728038131 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204e447640 1 --1- 172.21.15.146:0/3728038131 >> v1:172.21.15.146:6789/0 conn(0x7f2048163d80 0x7f2048164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48192/0 (socket says 172.21.15.146:48192) 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204d445640 1 --2- 172.21.15.146:0/3728038131 >> v2:172.21.15.177:3300/0 conn(0x7f20480d0020 0x7f20480d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 -- 172.21.15.146:0/3728038131 >> v2:172.21.15.177:3300/0 conn(0x7f20480d0020 msgr2=0x7f20480d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:50.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 --2- 172.21.15.146:0/3728038131 >> v2:172.21.15.177:3300/0 conn(0x7f20480d0020 0x7f20480d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 -- 172.21.15.146:0/3728038131 >> v1:172.21.15.146:6789/0 conn(0x7f2048163d80 legacy=0x7f2048164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204dc46640 1 -- 172.21.15.146:0/3728038131 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20480d0970 con 0x7f20480d4f60 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.988+0000 7f204cc44640 1 -- 172.21.15.146:0/3728038131 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3321381155 0 0) 0x7f2040001490 con 0x7f2048163d80 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.989+0000 7f204dc46640 1 --2- 172.21.15.146:0/3728038131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d4f60 0x7f20480d5370 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f20480b8c90 tx=0x7f204402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ede21d317bf15e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.989+0000 7f204cc44640 1 -- 172.21.15.146:0/3728038131 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044037070 con 0x7f20480d4f60 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.989+0000 7f204cc44640 1 -- 172.21.15.146:0/3728038131 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2044032440 con 0x7f20480d4f60 2024-04-25T15:22:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.989+0000 7f204cc44640 1 -- 172.21.15.146:0/3728038131 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044032810 con 0x7f20480d4f60 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 -- 172.21.15.146:0/3728038131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d4f60 msgr2=0x7f20480d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 --2- 172.21.15.146:0/3728038131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d4f60 0x7f20480d5370 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f20480b8c90 tx=0x7f204402a150 comp rx=0 tx=0).stop 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 -- 172.21.15.146:0/3728038131 shutdown_connections 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 --2- 172.21.15.146:0/3728038131 >> v2:172.21.15.177:3300/0 conn(0x7f20480d0020 0x7f20480d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 --2- 172.21.15.146:0/3728038131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d4f60 0x7f20480d5370 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 -- 172.21.15.146:0/3728038131 >> 172.21.15.146:0/3728038131 conn(0x7f20480d2450 msgr2=0x7f20480d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 -- 172.21.15.146:0/3728038131 shutdown_connections 2024-04-25T15:22:50.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.990+0000 7f204feaa640 1 -- 172.21.15.146:0/3728038131 wait complete. 2024-04-25T15:22:50.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.993+0000 7f204feaa640 1 Processor -- start 2024-04-25T15:22:50.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.993+0000 7f204feaa640 1 -- start start 2024-04-25T15:22:50.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204feaa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 0x7f2048161520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:50.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204feaa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20480d4f60 0x7f204814dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204feaa640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2048163d80 0x7f204806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204feaa640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20480d15f0 con 0x7f20480d0020 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204feaa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20480d1070 con 0x7f20480d4f60 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204feaa640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f20480b5150 con 0x7f2048163d80 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 0x7f2048161520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 0x7f2048161520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58364/0 (socket says 172.21.15.146:58364) 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 -- 172.21.15.146:0/1231734931 learned_addr learned my addr 172.21.15.146:0/1231734931 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204e447640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2048163d80 0x7f204806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204d445640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20480d4f60 0x7f204814dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:50.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 -- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2048163d80 msgr2=0x7f204806d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2048163d80 0x7f204806d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 -- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20480d4f60 msgr2=0x7f204814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20480d4f60 0x7f204814dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.995+0000 7f204dc46640 1 -- 172.21.15.146:0/1231734931 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2048162650 con 0x7f20480d0020 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.996+0000 7f204dc46640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 0x7f2048161520 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f20440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.996+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044041020 con 0x7f20480d0020 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.996+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20440306c0 con 0x7f20480d0020 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.996+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044037040 con 0x7f20480d0020 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.996+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2048157760 con 0x7f20480d0020 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.996+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2048162370 con 0x7f20480d0020 2024-04-25T15:22:51.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.997+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f204806a0e0 con 0x7f20480d0020 2024-04-25T15:22:51.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.998+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f2044044020 con 0x7f20480d0020 2024-04-25T15:22:51.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.998+0000 7f203e7fc640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f202c0745d0 0x7f202c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:51.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.999+0000 7f204d445640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f202c0745d0 0x7f202c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:51.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.999+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f20440b6050 con 0x7f20480d0020 2024-04-25T15:22:51.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:50.999+0000 7f204d445640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f202c0745d0 0x7f202c076ae0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f20480cf730 tx=0x7f2038008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:51.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.002+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f204404b030 con 0x7f20480d0020 2024-04-25T15:22:51.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.237+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20480cfa90 con 0x7f20480d0020 2024-04-25T15:22:51.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.237+0000 7f203e7fc640 1 -- 172.21.15.146:0/1231734931 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+5184 (secure 0 0 0) 0x7f20480d0970 con 0x7f20480d0020 2024-04-25T15:22:51.239 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:51.239 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:50.845165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T15:22:51.239 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 33 2024-04-25T15:22:51.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f202c0745d0 msgr2=0x7f202c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f202c0745d0 0x7f202c076ae0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f20480cf730 tx=0x7f2038008040 comp rx=0 tx=0).stop 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 msgr2=0x7f2048161520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 0x7f2048161520 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f20440046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 shutdown_connections 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f202c0745d0 0x7f202c076ae0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2048163d80 0x7f204806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20480d4f60 0x7f204814dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 --2- 172.21.15.146:0/1231734931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20480d0020 0x7f2048161520 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 >> 172.21.15.146:0/1231734931 conn(0x7f20480d2450 msgr2=0x7f204815f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 shutdown_connections 2024-04-25T15:22:51.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:51.240+0000 7f204feaa640 1 -- 172.21.15.146:0/1231734931 wait complete. 2024-04-25T15:22:51.658 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:22:19.583041+0000', 'modified': '2024-04-25T15:22:50.845165+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6819}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6819': {'gid': 6819, 'name': 'c', 'rank': 0, 'incarnation': 33, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.146:6836/2021557185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 2021557185}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 2021557185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T15:22:51.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6819:up:replay 2024-04-25T15:22:52.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:52.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.778+0000 7fd7b87b2640 1 Processor -- start 2024-04-25T15:22:52.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.778+0000 7fd7b87b2640 1 -- start start 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b87b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7b00715e0 0x7fd7b006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b87b2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b0070190 0x7fd7b0169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b87b2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b00705a0 con 0x7fd7b00b7350 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b87b2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b0071bc0 con 0x7fd7b00715e0 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b87b2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b00d1b00 con 0x7fd7b0070190 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b654e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd7b00b7350 0x7fd7b0160000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48194/0 (socket says 172.21.15.146:48194) 2024-04-25T15:22:52.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b5d4d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7b00715e0 0x7fd7b006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b654e640 1 -- 172.21.15.146:0/2071825904 learned_addr learned my addr 172.21.15.146:0/2071825904 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b6d4f640 1 --2- 172.21.15.146:0/2071825904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b0070190 0x7fd7b0169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b5d4d640 1 -- 172.21.15.146:0/2071825904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b0070190 msgr2=0x7fd7b0169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b5d4d640 1 --2- 172.21.15.146:0/2071825904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b0070190 0x7fd7b0169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b5d4d640 1 -- 172.21.15.146:0/2071825904 >> v1:172.21.15.146:6789/0 conn(0x7fd7b00b7350 legacy=0x7fd7b0160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b5d4d640 1 -- 172.21.15.146:0/2071825904 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b00d1de0 con 0x7fd7b00715e0 2024-04-25T15:22:52.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b554c640 1 -- 172.21.15.146:0/2071825904 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547337136 0 0) 0x7fd7ac001110 con 0x7fd7b00b7350 2024-04-25T15:22:52.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b5d4d640 1 --2- 172.21.15.146:0/2071825904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b00715e0 0x7fd7b006fc50 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00097a0 tx=0x7fd7a002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535d3c2dcec04b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:52.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b554c640 1 -- 172.21.15.146:0/2071825904 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0037070 con 0x7fd7b00715e0 2024-04-25T15:22:52.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b554c640 1 -- 172.21.15.146:0/2071825904 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a0032440 con 0x7fd7b00715e0 2024-04-25T15:22:52.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.781+0000 7fd7b554c640 1 -- 172.21.15.146:0/2071825904 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0032810 con 0x7fd7b00715e0 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 -- 172.21.15.146:0/2071825904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b00715e0 msgr2=0x7fd7b006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/2071825904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b00715e0 0x7fd7b006fc50 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00097a0 tx=0x7fd7a002a180 comp rx=0 tx=0).stop 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 -- 172.21.15.146:0/2071825904 shutdown_connections 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/2071825904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b0070190 0x7fd7b0169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/2071825904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b00715e0 0x7fd7b006fc50 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 -- 172.21.15.146:0/2071825904 >> 172.21.15.146:0/2071825904 conn(0x7fd7b009b2b0 msgr2=0x7fd7b00b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 -- 172.21.15.146:0/2071825904 shutdown_connections 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.782+0000 7fd7b87b2640 1 -- 172.21.15.146:0/2071825904 wait complete. 2024-04-25T15:22:52.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.784+0000 7fd7b87b2640 1 Processor -- start 2024-04-25T15:22:52.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.784+0000 7fd7b87b2640 1 -- start start 2024-04-25T15:22:52.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b87b2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 0x7fd7b014a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b87b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7b00715e0 0x7fd7b01fa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b87b2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b00b7350 0x7fd7b01487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b87b2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b00cf4b0 con 0x7fd7b0070190 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b87b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b00d23a0 con 0x7fd7b00715e0 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b87b2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b0070980 con 0x7fd7b00b7350 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 0x7fd7b014a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 0x7fd7b014a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58378/0 (socket says 172.21.15.146:58378) 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 -- 172.21.15.146:0/312376128 learned_addr learned my addr 172.21.15.146:0/312376128 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 -- 172.21.15.146:0/312376128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b00b7350 msgr2=0x7fd7b01487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b00b7350 0x7fd7b01487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 -- 172.21.15.146:0/312376128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7b00715e0 msgr2=0x7fd7b01fa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7b00715e0 0x7fd7b01fa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:52.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 -- 172.21.15.146:0/312376128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b014b790 con 0x7fd7b0070190 2024-04-25T15:22:52.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.785+0000 7fd7b654e640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 0x7fd7b014a860 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00c410 tx=0x7fd7ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:52.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.786+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0074e0 con 0x7fd7b0070190 2024-04-25T15:22:52.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.786+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac0047e0 con 0x7fd7b0070190 2024-04-25T15:22:52.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.786+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b014b1d0 con 0x7fd7b0070190 2024-04-25T15:22:52.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.786+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac004bd0 con 0x7fd7b0070190 2024-04-25T15:22:52.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.786+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b01f9a90 con 0x7fd7b0070190 2024-04-25T15:22:52.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.787+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b006a0e0 con 0x7fd7b0070190 2024-04-25T15:22:52.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.787+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fd7ac011070 con 0x7fd7b0070190 2024-04-25T15:22:52.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.788+0000 7fd7a6ffd640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd7940745d0 0x7fd794076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:52.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.788+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fd7ac0951c0 con 0x7fd7b0070190 2024-04-25T15:22:52.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.788+0000 7fd7b5d4d640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd7940745d0 0x7fd794076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:52.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.788+0000 7fd7b5d4d640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd7940745d0 0x7fd794076ae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0034000 tx=0x7fd7a002c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:52.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:52.793+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7b00705a0 con 0x7fd7b0070190 2024-04-25T15:22:53.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.009+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7b00711b0 con 0x7fd7b0070190 2024-04-25T15:22:53.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.009+0000 7fd7a6ffd640 1 -- 172.21.15.146:0/312376128 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5184 (secure 0 0 0) 0x7fd7ac05f6b0 con 0x7fd7b0070190 2024-04-25T15:22:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:53.011 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:51.904264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T15:22:53.012 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 35 2024-04-25T15:22:53.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.011+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd7940745d0 msgr2=0x7fd794076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:53.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.011+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd7940745d0 0x7fd794076ae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0034000 tx=0x7fd7a002c000 comp rx=0 tx=0).stop 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.011+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 msgr2=0x7fd7b014a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.011+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 0x7fd7b014a860 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00c410 tx=0x7fd7ac00c8d0 comp rx=0 tx=0).stop 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.011+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 shutdown_connections 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.011+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd7940745d0 0x7fd794076ae0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.012+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7b00b7350 0x7fd7b01487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.012+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7b00715e0 0x7fd7b01fa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.012+0000 7fd7b87b2640 1 --2- 172.21.15.146:0/312376128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7b0070190 0x7fd7b014a860 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.012+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 >> 172.21.15.146:0/312376128 conn(0x7fd7b009b2b0 msgr2=0x7fd7b00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.012+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 shutdown_connections 2024-04-25T15:22:53.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:53.012+0000 7fd7b87b2640 1 -- 172.21.15.146:0/312376128 wait complete. 2024-04-25T15:22:53.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:22:19.583041+0000', 'modified': '2024-04-25T15:22:51.904264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6819}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6819': {'gid': 6819, 'name': 'c', 'rank': 0, 'incarnation': 33, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.146:6836/2021557185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 2021557185}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 2021557185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T15:22:53.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6819:up:rejoin 2024-04-25T15:22:54.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:22:54.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.556+0000 7f650df34640 1 Processor -- start 2024-04-25T15:22:54.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.556+0000 7f650df34640 1 -- start start 2024-04-25T15:22:54.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f650df34640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65080d0730 0x7f65080d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:54.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f650df34640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65080b9200 con 0x7f65080d4f60 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f650df34640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65080d1070 con 0x7f650806ccc0 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f650df34640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f650806c550 con 0x7f65080d0730 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f6506ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65080d0730 0x7f65080d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f65077fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f65080d4f60 0x7f65080d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48208/0 (socket says 172.21.15.146:48208) 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.557+0000 7f65077fe640 1 -- 172.21.15.146:0/2400689204 learned_addr learned my addr 172.21.15.146:0/2400689204 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.558+0000 7f6506ffd640 1 -- 172.21.15.146:0/2400689204 >> v1:172.21.15.177:6789/0 conn(0x7f650806ccc0 legacy=0x7f650806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.558+0000 7f6506ffd640 1 -- 172.21.15.146:0/2400689204 >> v1:172.21.15.146:6789/0 conn(0x7f65080d4f60 legacy=0x7f65080d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.558+0000 7f65067fc640 1 -- 172.21.15.146:0/2400689204 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565125050 0 0) 0x7f64f8001490 con 0x7f65080d4f60 2024-04-25T15:22:54.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.558+0000 7f6506ffd640 1 -- 172.21.15.146:0/2400689204 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65080d1310 con 0x7f65080d0730 2024-04-25T15:22:54.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.559+0000 7f6506ffd640 1 --2- 172.21.15.146:0/2400689204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65080d0730 0x7f65080d5370 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0097a0 tx=0x7f64fc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc71e3ea00b7902 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:54.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.559+0000 7f65067fc640 1 -- 172.21.15.146:0/2400689204 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc037070 con 0x7f65080d0730 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.559+0000 7f65067fc640 1 -- 172.21.15.146:0/2400689204 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64fc032440 con 0x7f65080d0730 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.559+0000 7f65067fc640 1 -- 172.21.15.146:0/2400689204 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc02ae60 con 0x7f65080d0730 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.559+0000 7f650df34640 1 -- 172.21.15.146:0/2400689204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65080d0730 msgr2=0x7f65080d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.559+0000 7f650df34640 1 --2- 172.21.15.146:0/2400689204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65080d0730 0x7f65080d5370 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0097a0 tx=0x7f64fc02a110 comp rx=0 tx=0).stop 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.560+0000 7f650df34640 1 -- 172.21.15.146:0/2400689204 shutdown_connections 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.560+0000 7f650df34640 1 --2- 172.21.15.146:0/2400689204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65080d0730 0x7f65080d5370 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.560+0000 7f650df34640 1 -- 172.21.15.146:0/2400689204 >> 172.21.15.146:0/2400689204 conn(0x7f65080d2450 msgr2=0x7f65080d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.560+0000 7f650df34640 1 -- 172.21.15.146:0/2400689204 shutdown_connections 2024-04-25T15:22:54.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.560+0000 7f650df34640 1 -- 172.21.15.146:0/2400689204 wait complete. 2024-04-25T15:22:54.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.562+0000 7f650df34640 1 Processor -- start 2024-04-25T15:22:54.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.563+0000 7f650df34640 1 -- start start 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f650df34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f650806ccc0 0x7f65081630f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f650df34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65080d4f60 0x7f650815c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f650df34640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 0x7f650815c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f650df34640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f65080b5150 con 0x7f650814e6f0 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f650df34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f650806c830 con 0x7f65080d4f60 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f650df34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f65080b52d0 con 0x7f650806ccc0 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 0x7f650815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:54.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 0x7f650815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58404/0 (socket says 172.21.15.146:58404) 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 -- 172.21.15.146:0/2505897185 learned_addr learned my addr 172.21.15.146:0/2505897185 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f65077fe640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f650806ccc0 0x7f65081630f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6506ffd640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65080d4f60 0x7f650815c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 -- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f650806ccc0 msgr2=0x7f65081630f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f650806ccc0 0x7f65081630f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 -- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65080d4f60 msgr2=0x7f650815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65080d4f60 0x7f650815c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.565+0000 7f6507fff640 1 -- 172.21.15.146:0/2505897185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650815e800 con 0x7f650814e6f0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.566+0000 7f6507fff640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 0x7f650815c870 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f650000c3f0 tx=0x7f650000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.566+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500017070 con 0x7f650814e6f0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.566+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65000052d0 con 0x7f650814e6f0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.566+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500003830 con 0x7f650814e6f0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.566+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650815eae0 con 0x7f650814e6f0 2024-04-25T15:22:54.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.566+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650815f0a0 con 0x7f650814e6f0 2024-04-25T15:22:54.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.567+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f650806a0e0 con 0x7f650814e6f0 2024-04-25T15:22:54.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.568+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f650001a070 con 0x7f650814e6f0 2024-04-25T15:22:54.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.569+0000 7f64effff640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f64f40745d0 0x7f64f4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:54.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.569+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f65000947e0 con 0x7f650814e6f0 2024-04-25T15:22:54.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.569+0000 7f65077fe640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f64f40745d0 0x7f64f4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:54.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.569+0000 7f65077fe640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f64f40745d0 0x7f64f4076ae0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f64f80012e0 tx=0x7f64f8005bf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:54.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.578+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f650005ee20 con 0x7f650814e6f0 2024-04-25T15:22:54.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.851+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6508163a50 con 0x7f650814e6f0 2024-04-25T15:22:54.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.852+0000 7f64effff640 1 -- 172.21.15.146:0/2505897185 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5191 (secure 0 0 0) 0x7f650005e720 con 0x7f650814e6f0 2024-04-25T15:22:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:54.853 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:53.856951+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6819,"qdb_cluster":[6819]},"id":3}]} 2024-04-25T15:22:54.853 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 36 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f64f40745d0 msgr2=0x7f64f4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f64f40745d0 0x7f64f4076ae0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f64f80012e0 tx=0x7f64f8005bf0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 msgr2=0x7f650815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 0x7f650815c870 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f650000c3f0 tx=0x7f650000c7f0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 shutdown_connections 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f64f40745d0 0x7f64f4076ae0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f650814e6f0 0x7f650815c870 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65080d4f60 0x7f650815c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 --2- 172.21.15.146:0/2505897185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f650806ccc0 0x7f65081630f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:54.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 >> 172.21.15.146:0/2505897185 conn(0x7f65080d2450 msgr2=0x7f650806d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:54.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 shutdown_connections 2024-04-25T15:22:54.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:54.854+0000 7f650df34640 1 -- 172.21.15.146:0/2505897185 wait complete. 2024-04-25T15:22:55.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:22:19.583041+0000', 'modified': '2024-04-25T15:22:53.856951+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6819}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6819': {'gid': 6819, 'name': 'c', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6836/2021557185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 2021557185}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 2021557185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6819, 'qdb_cluster': [6819]} 2024-04-25T15:22:55.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:22:55.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T15:22:55.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.387+0000 7f6630c4c640 1 Processor -- start 2024-04-25T15:22:55.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.387+0000 7f6630c4c640 1 -- start start 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f6630c4c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f662c0b4b10 0x7f662c0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f6630c4c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f662c0cfa90 0x7f662c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f6630c4c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f662c0cfea0 con 0x7f662c0d2c50 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f6630c4c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f662c0b4f20 con 0x7f662c0b4b10 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f6630c4c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f662c0b3a70 con 0x7f662c0cfa90 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f662c0b4b10 0x7f662c0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f662c0b4b10 0x7f662c0cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58412/0 (socket says 172.21.15.146:58412) 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 -- 172.21.15.146:0/2272877234 learned_addr learned my addr 172.21.15.146:0/2272877234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:55.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f6629d9b640 1 --2- 172.21.15.146:0/2272877234 >> v2:172.21.15.177:3300/0 conn(0x7f662c0cfa90 0x7f662c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 -- 172.21.15.146:0/2272877234 >> v2:172.21.15.177:3300/0 conn(0x7f662c0cfa90 msgr2=0x7f662c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 --2- 172.21.15.146:0/2272877234 >> v2:172.21.15.177:3300/0 conn(0x7f662c0cfa90 0x7f662c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 -- 172.21.15.146:0/2272877234 >> v1:172.21.15.177:6789/0 conn(0x7f662c0d2c50 legacy=0x7f662c0af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.389+0000 7f662a59c640 1 -- 172.21.15.146:0/2272877234 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c0b3d50 con 0x7f662c0b4b10 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.390+0000 7f662a59c640 1 --2- 172.21.15.146:0/2272877234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cf550 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f662c0d24c0 tx=0x7f661c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663d868609a125da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.390+0000 7f662959a640 1 -- 172.21.15.146:0/2272877234 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c004030 con 0x7f662c0b4b10 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.390+0000 7f662959a640 1 -- 172.21.15.146:0/2272877234 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f661c0041d0 con 0x7f662c0b4b10 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.390+0000 7f662959a640 1 -- 172.21.15.146:0/2272877234 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c003610 con 0x7f662c0b4b10 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 -- 172.21.15.146:0/2272877234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 msgr2=0x7f662c0cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 --2- 172.21.15.146:0/2272877234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cf550 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f662c0d24c0 tx=0x7f661c02a180 comp rx=0 tx=0).stop 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 -- 172.21.15.146:0/2272877234 shutdown_connections 2024-04-25T15:22:55.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 --2- 172.21.15.146:0/2272877234 >> v2:172.21.15.177:3300/0 conn(0x7f662c0cfa90 0x7f662c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 --2- 172.21.15.146:0/2272877234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cf550 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 -- 172.21.15.146:0/2272877234 >> 172.21.15.146:0/2272877234 conn(0x7f662c0b73b0 msgr2=0x7f662c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:55.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 -- 172.21.15.146:0/2272877234 shutdown_connections 2024-04-25T15:22:55.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.391+0000 7f6630c4c640 1 -- 172.21.15.146:0/2272877234 wait complete. 2024-04-25T15:22:55.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.394+0000 7f6630c4c640 1 Processor -- start 2024-04-25T15:22:55.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.394+0000 7f6630c4c640 1 -- start start 2024-04-25T15:22:55.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6630c4c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6630c4c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f662c0cfa90 0x7f662c150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6630c4c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f662c0d2c50 0x7f662c15fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6630c4c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f662c0b45f0 con 0x7f662c0b4b10 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6630c4c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f662c0b4310 con 0x7f662c0d2c50 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6630c4c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f662c0d3060 con 0x7f662c0cfa90 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58428/0 (socket says 172.21.15.146:58428) 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 -- 172.21.15.146:0/322708185 learned_addr learned my addr 172.21.15.146:0/322708185 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f6629d9b640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f662c0cfa90 0x7f662c150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662ad9d640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f662c0d2c50 0x7f662c15fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 -- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f662c0cfa90 msgr2=0x7f662c150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f662c0cfa90 0x7f662c150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 -- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f662c0d2c50 msgr2=0x7f662c15fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f662c0d2c50 0x7f662c15fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.396+0000 7f662a59c640 1 -- 172.21.15.146:0/322708185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c151360 con 0x7f662c0b4b10 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.397+0000 7f662a59c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cc0a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f661c000c00 tx=0x7f661c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.397+0000 7f661affd640 1 -- 172.21.15.146:0/322708185 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c041020 con 0x7f662c0b4b10 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.397+0000 7f661affd640 1 -- 172.21.15.146:0/322708185 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f661c003c50 con 0x7f662c0b4b10 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.397+0000 7f661affd640 1 -- 172.21.15.146:0/322708185 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c004030 con 0x7f662c0b4b10 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.397+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c0c7560 con 0x7f662c0b4b10 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.397+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c0c7b20 con 0x7f662c0b4b10 2024-04-25T15:22:55.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.399+0000 7f661affd640 1 -- 172.21.15.146:0/322708185 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f661c044020 con 0x7f662c0b4b10 2024-04-25T15:22:55.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.400+0000 7f661affd640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f660c0745d0 0x7f660c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.400+0000 7f661affd640 1 -- 172.21.15.146:0/322708185 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f661c0b61c0 con 0x7f662c0b4b10 2024-04-25T15:22:55.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.400+0000 7f6629d9b640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f660c0745d0 0x7f660c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.400+0000 7f6629d9b640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f660c0745d0 0x7f660c076ae0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f662c1544f0 tx=0x7f6620008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.401+0000 7f6637bda740 1 Processor -- start 2024-04-25T15:22:55.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.402+0000 7f6637bda740 1 -- start start 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d738e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6637bda740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 0x55d43d7357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6637bda740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d43d66a740 con 0x55d43d33b3b0 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d43d6063a0 con 0x55d43d364270 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55d43d666700 con 0x55d43d343590 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.403+0000 7f6629d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 0x55d43d7357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 0x55d43d7357b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58432/0 (socket says 172.21.15.146:58432) 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f662a59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d738e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 -- 172.21.15.146:0/188951473 learned_addr learned my addr 172.21.15.146:0/188951473 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:55.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f662ad9d640 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 -- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 msgr2=0x55d43d70ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 -- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 msgr2=0x55d43d738e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d738e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 -- 172.21.15.146:0/188951473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d43d622750 con 0x55d43d33b3b0 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f662a59c640 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d738e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f662ad9d640 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f6629d9b640 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 0x55d43d7357b0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6620004440 tx=0x7f6620012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432e6ed4ae24126f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.404+0000 7f660bfff640 1 -- 172.21.15.146:0/188951473 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620019070 con 0x55d43d33b3b0 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.405+0000 7f660bfff640 1 -- 172.21.15.146:0/188951473 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6620025070 con 0x55d43d33b3b0 2024-04-25T15:22:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.405+0000 7f660bfff640 1 -- 172.21.15.146:0/188951473 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620020470 con 0x55d43d33b3b0 2024-04-25T15:22:55.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 -- 172.21.15.146:0/188951473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 msgr2=0x55d43d7357b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 0x55d43d7357b0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6620004440 tx=0x7f6620012000 comp rx=0 tx=0).stop 2024-04-25T15:22:55.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 -- 172.21.15.146:0/188951473 shutdown_connections 2024-04-25T15:22:55.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d33b3b0 0x55d43d7357b0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 --2- 172.21.15.146:0/188951473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d738e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 -- 172.21.15.146:0/188951473 >> 172.21.15.146:0/188951473 conn(0x55d43d3583c0 msgr2=0x55d43d70e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 -- 172.21.15.146:0/188951473 shutdown_connections 2024-04-25T15:22:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.406+0000 7f6637bda740 1 -- 172.21.15.146:0/188951473 wait complete. 2024-04-25T15:22:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.407+0000 7f6637bda740 1 -- start start 2024-04-25T15:22:55.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.407+0000 7f6637bda740 1 Processor -- start 2024-04-25T15:22:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d73b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d343590 0x55d43d762c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d7631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d43d5f1fe0 con 0x55d43d364270 2024-04-25T15:22:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d43d763b20 con 0x55d43d343590 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55d43d739e00 con 0x55d43d33b3b0 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f662ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d7631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f662ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d7631d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58436/0 (socket says 172.21.15.146:58436) 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f662ad9d640 1 -- 172.21.15.146:0/3988659921 learned_addr learned my addr 172.21.15.146:0/3988659921 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f662a59c640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d73b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6637bda740 10 client.0 _ll_register_callbacks cb 0x55d43d608150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:22:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.410+0000 7f6629d9b640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d343590 0x55d43d762c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f662ad9d640 1 -- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 msgr2=0x55d43d73b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f662ad9d640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d73b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f662ad9d640 1 -- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d343590 msgr2=0x55d43d762c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f662ad9d640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d343590 0x55d43d762c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f662ad9d640 1 -- 172.21.15.146:0/3988659921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d43d37c2d0 con 0x55d43d364270 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f662ad9d640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d7631d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f662400ab90 tx=0x7f6624011580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7f64e20d7610d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f66097fa640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f66097fa640 1 -- 172.21.15.146:0/3988659921 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662401a070 con 0x55d43d364270 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f66097fa640 1 -- 172.21.15.146:0/3988659921 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f6624011e30 con 0x55d43d364270 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.411+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6620025070 con 0x55d43d364270 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f66097fa640 1 -- 172.21.15.146:0/3988659921 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6624025050 con 0x55d43d364270 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 10 client.6864 fetch_fsmap learned FSMap version 36 2024-04-25T15:22:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x55d43d4466b0 con 0x55d43d364270 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f66097fa640 1 -- 172.21.15.146:0/3988659921 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6620025070 con 0x55d43d364270 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f66097fa640 1 -- 172.21.15.146:0/3988659921 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f6624028020 con 0x55d43d364270 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 10 client.6864 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 10 client.6864 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 20 client.6864 populate_metadata read hostname 'smithi146' 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d322e40 0x55d43d7709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 4 client.6864 mds_command: new command op to 6819 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.412+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 --> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d43d678350 con 0x55d43d322e40 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.413+0000 7f662a59c640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d322e40 0x55d43d7709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.413+0000 7f66097fa640 10 client.6864 ms_handle_connect on v2:172.21.15.146:6834/2021557185 2024-04-25T15:22:55.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.413+0000 7f662a59c640 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d322e40 0x55d43d7709d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdee427252e29c9c server_cookie=5f3ed78bb0785ac4 in_seq=0 out_seq=0 2024-04-25T15:22:55.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f66097fa640 1 -- 172.21.15.146:0/3988659921 <== mds.0 v2:172.21.15.146:6834/2021557185 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d43d678350 con 0x55d43d322e40 2024-04-25T15:22:55.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f66097fa640 10 client.6864 handle_command_reply: tid=0 2024-04-25T15:22:55.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 client.6864 shutdown 2024-04-25T15:22:55.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 msgr2=0x55d43d7631d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d7631d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f662400ab90 tx=0x7f6624011580 comp rx=0 tx=0).stop 2024-04-25T15:22:55.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 shutdown_connections 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d322e40 0x55d43d7709d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d7631d0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d343590 0x55d43d762c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f66097fa640 0 client.6864 ms_handle_reset on v2:172.21.15.146:6834/2021557185 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 --2- 172.21.15.146:0/3988659921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d73b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.415+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 >> 172.21.15.146:0/3988659921 conn(0x55d43d3583c0 msgr2=0x55d43d70e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.416+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 shutdown_connections 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.416+0000 7f6637bda740 1 -- 172.21.15.146:0/3988659921 wait complete. 2024-04-25T15:22:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.416+0000 7f6637bda740 20 client.6864 trim_cache size 0 max 16384 2024-04-25T15:22:55.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.449+0000 7f6637bda740 1 Processor -- start 2024-04-25T15:22:55.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.449+0000 7f6637bda740 1 -- start start 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d33b3b0 0x55d43d738e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6637bda740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d710140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6637bda740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f662c0d1d60 con 0x55d43d364270 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f662c0c9010 con 0x55d43d33b3b0 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f662c0d1710 con 0x55d43d343590 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f662ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d710140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f662ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d710140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58444/0 (socket says 172.21.15.146:58444) 2024-04-25T15:22:55.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f662ad9d640 1 -- 172.21.15.146:0/3626623572 learned_addr learned my addr 172.21.15.146:0/3626623572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:55.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.451+0000 7f6629d9b640 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f6629d9b640 1 -- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d33b3b0 msgr2=0x55d43d738e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f6629d9b640 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d33b3b0 0x55d43d738e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f6629d9b640 1 -- 172.21.15.146:0/3626623572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 msgr2=0x55d43d710140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f6629d9b640 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d710140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f6629d9b640 1 -- 172.21.15.146:0/3626623572 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d43d7393a0 con 0x55d43d343590 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f662ad9d640 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d710140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f6629d9b640 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70fc00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f662000a400 tx=0x7f6620014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdbeb90a11d4fe6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f660b7fe640 1 -- 172.21.15.146:0/3626623572 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620019040 con 0x55d43d343590 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f660b7fe640 1 -- 172.21.15.146:0/3626623572 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6620025040 con 0x55d43d343590 2024-04-25T15:22:55.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.452+0000 7f660b7fe640 1 -- 172.21.15.146:0/3626623572 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620004100 con 0x55d43d343590 2024-04-25T15:22:55.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 -- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 msgr2=0x55d43d70fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70fc00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f662000a400 tx=0x7f6620014000 comp rx=0 tx=0).stop 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 -- 172.21.15.146:0/3626623572 shutdown_connections 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d364270 0x55d43d710140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d343590 0x55d43d70fc00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 --2- 172.21.15.146:0/3626623572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d33b3b0 0x55d43d738e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.454+0000 7f6637bda740 1 -- 172.21.15.146:0/3626623572 >> 172.21.15.146:0/3626623572 conn(0x55d43d3583c0 msgr2=0x55d43d756280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:55.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.455+0000 7f6637bda740 1 -- 172.21.15.146:0/3626623572 shutdown_connections 2024-04-25T15:22:55.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.455+0000 7f6637bda740 1 -- 172.21.15.146:0/3626623572 wait complete. 2024-04-25T15:22:55.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.457+0000 7f6637bda740 1 -- start start 2024-04-25T15:22:55.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.457+0000 7f6637bda740 1 Processor -- start 2024-04-25T15:22:55.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d7492a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 0x55d43d7497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d7d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d43d7518e0 con 0x55d43d343590 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d43d763860 con 0x55d43d364270 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x55d43d751f00 con 0x55d43d33b3b0 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6637bda740 10 client.0 _ll_register_callbacks cb 0x7f6624009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6629d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 0x55d43d7497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6629d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 0x55d43d7497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58454/0 (socket says 172.21.15.146:58454) 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f6629d9b640 1 -- 172.21.15.146:0/2340653334 learned_addr learned my addr 172.21.15.146:0/2340653334 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.459+0000 7f662a59c640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d7492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f662ad9d640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d7d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f6629d9b640 1 -- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 msgr2=0x55d43d7492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f6629d9b640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d7492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f6629d9b640 1 -- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 msgr2=0x55d43d7d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f6629d9b640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d7d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f6629d9b640 1 -- 172.21.15.146:0/2340653334 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d43d74ac80 con 0x55d43d343590 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f6629d9b640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 0x55d43d7497e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6620021000 tx=0x7f6620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfe16c8111e5199 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:55.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f66097fa640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.460+0000 7f66097fa640 1 -- 172.21.15.146:0/2340653334 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662003c070 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f66097fa640 1 -- 172.21.15.146:0/2340653334 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f6620020c30 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6620025040 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f66097fa640 1 -- 172.21.15.146:0/2340653334 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6620047050 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 10 client.6870 fetch_fsmap learned FSMap version 36 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x55d43d735eb0 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f66097fa640 1 -- 172.21.15.146:0/2340653334 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6620025040 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f66097fa640 1 -- 172.21.15.146:0/2340653334 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 36) ==== 3494+0+0 (secure 0 0 0) 0x7f662004a020 con 0x55d43d343590 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 10 client.6870 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T15:22:55.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 10 client.6870 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 20 client.6870 populate_metadata read hostname 'smithi146' 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d7d1bc0 0x55d43d73b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 4 client.6870 mds_command: new command op to 6819 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 --> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] -- command(tid 0: {"prefix": "status"}) -- 0x55d43d6721c0 con 0x55d43d7d1bc0 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.461+0000 7f662a59c640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d7d1bc0 0x55d43d73b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.462+0000 7f66097fa640 10 client.6870 ms_handle_connect on v2:172.21.15.146:6834/2021557185 2024-04-25T15:22:55.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.462+0000 7f662a59c640 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d7d1bc0 0x55d43d73b980 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b51a1fe637e2f316 server_cookie=1391ed7e54b6b7cb in_seq=0 out_seq=0 2024-04-25T15:22:55.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f66097fa640 1 -- 172.21.15.146:0/2340653334 <== mds.0 v2:172.21.15.146:6834/2021557185 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d43d6721c0 con 0x55d43d7d1bc0 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f66097fa640 10 client.6870 handle_command_reply: tid=0 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 client.6870 shutdown 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 msgr2=0x55d43d7497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 0x55d43d7497e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6620021000 tx=0x7f6620004730 comp rx=0 tx=0).stop 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 shutdown_connections 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:6834/2021557185,v1:172.21.15.146:6836/2021557185] conn(0x55d43d7d1bc0 0x55d43d73b980 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d43d364270 0x55d43d7d1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d43d343590 0x55d43d7497e0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f66097fa640 0 client.6870 ms_handle_reset on v2:172.21.15.146:6834/2021557185 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 --2- 172.21.15.146:0/2340653334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55d43d33b3b0 0x55d43d7492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 >> 172.21.15.146:0/2340653334 conn(0x55d43d3583c0 msgr2=0x55d43d708ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 shutdown_connections 2024-04-25T15:22:55.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 1 -- 172.21.15.146:0/2340653334 wait complete. 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.464+0000 7f6637bda740 20 client.6870 trim_cache size 0 max 16384 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "c6e9a678-3722-48f4-a297-70f509c13e29", 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6819, 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-25T15:22:55.470 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-25T15:22:55.471 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 4.6141301769999998, 2024-04-25T15:22:55.471 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 36, 2024-04-25T15:22:55.471 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 57, 2024-04-25T15:22:55.471 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 57, 2024-04-25T15:22:55.471 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 5.5042305929999999 2024-04-25T15:22:55.471 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:22:55.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.465+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f660c0745d0 msgr2=0x7f660c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.465+0000 7f6630c4c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f660c0745d0 0x7f660c076ae0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f662c1544f0 tx=0x7f6620008040 comp rx=0 tx=0).stop 2024-04-25T15:22:55.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.465+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 msgr2=0x7f662c0cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:55.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.465+0000 7f6630c4c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cc0a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f661c000c00 tx=0x7f661c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 shutdown_connections 2024-04-25T15:22:55.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f660c0745d0 0x7f660c076ae0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f662c0d2c50 0x7f662c15fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f662c0cfa90 0x7f662c150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 --2- 172.21.15.146:0/322708185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662c0b4b10 0x7f662c0cc0a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:55.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 >> 172.21.15.146:0/322708185 conn(0x7f662c0b73b0 msgr2=0x7f662c06d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:55.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 shutdown_connections 2024-04-25T15:22:55.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55.466+0000 7f6630c4c640 1 -- 172.21.15.146:0/322708185 wait complete. 2024-04-25T15:22:55.487 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:22:55.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:22:55.535 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:22:55.576 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:22:55.576 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:22:55.576 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--timeout=1', '--', 'bash', '-c', Raw("'\n set -ex\n pushd `mktemp -d -p .`\n cp -a /usr .\n popd\n '")], check_status=True, hostname='smithi146') 2024-04-25T15:22:55.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:55 stdin-killer INFO: expiration expected; waiting 1 seconds for command to complete 2024-04-25T15:22:55.852 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-04-25T15:22:55.851+0000 7f1b09daf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:22:56.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56 stdin-killer ERROR: timeout expired: sending signal 9 to command and myself 2024-04-25T15:22:56.578 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 6: 59958 Killed ( cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:22:56.578 INFO:teuthology.orchestra.run.smithi146.stderr: set -ex 2024-04-25T15:22:56.578 INFO:teuthology.orchestra.run.smithi146.stderr: pushd `mktemp -d -p .` 2024-04-25T15:22:56.578 INFO:teuthology.orchestra.run.smithi146.stderr: cp -a /usr . 2024-04-25T15:22:56.578 INFO:teuthology.orchestra.run.smithi146.stderr: popd 2024-04-25T15:22:56.579 INFO:teuthology.orchestra.run.smithi146.stderr: ' ) 2024-04-25T15:22:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 137 2024-04-25T15:22:56.601 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:56.601 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:56.658 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:22:56.658 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:22:56.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:22:56.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.912+0000 7f936ed34640 1 Processor -- start 2024-04-25T15:22:56.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.912+0000 7f936ed34640 1 -- start start 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936ed34640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9368160e00 0x7f936815c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936ed34640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f936806cde0 0x7f936806d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936ed34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93680ccfb0 0x7f93680cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936ed34640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f936815c9d0 con 0x7f9368160e00 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936ed34640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f93681600d0 con 0x7f936806cde0 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936ed34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f93680b10b0 con 0x7f93680ccfb0 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9368160e00 0x7f936815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9368160e00 0x7f936815c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58468/0 (socket says 172.21.15.146:58468) 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 -- 172.21.15.146:0/4074594580 learned_addr learned my addr 172.21.15.146:0/4074594580 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f9367fff640 1 --2- 172.21.15.146:0/4074594580 >> v2:172.21.15.177:3300/0 conn(0x7f936806cde0 0x7f936806d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:56.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 -- 172.21.15.146:0/4074594580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93680ccfb0 msgr2=0x7f93680cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:56.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 --2- 172.21.15.146:0/4074594580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93680ccfb0 0x7f93680cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 -- 172.21.15.146:0/4074594580 >> v2:172.21.15.177:3300/0 conn(0x7f936806cde0 msgr2=0x7f936806d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:56.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 --2- 172.21.15.146:0/4074594580 >> v2:172.21.15.177:3300/0 conn(0x7f936806cde0 0x7f936806d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.914+0000 7f936cad0640 1 -- 172.21.15.146:0/4074594580 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936806d730 con 0x7f9368160e00 2024-04-25T15:22:56.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.915+0000 7f936cad0640 1 --2- 172.21.15.146:0/4074594580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368160e00 0x7f936815c490 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9368070570 tx=0x7f935c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12139ffdc48b8d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:56.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.915+0000 7f93677fe640 1 -- 172.21.15.146:0/4074594580 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c037070 con 0x7f9368160e00 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.915+0000 7f93677fe640 1 -- 172.21.15.146:0/4074594580 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f935c032440 con 0x7f9368160e00 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.915+0000 7f93677fe640 1 -- 172.21.15.146:0/4074594580 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c032810 con 0x7f9368160e00 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 -- 172.21.15.146:0/4074594580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368160e00 msgr2=0x7f936815c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 --2- 172.21.15.146:0/4074594580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368160e00 0x7f936815c490 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9368070570 tx=0x7f935c02a200 comp rx=0 tx=0).stop 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 -- 172.21.15.146:0/4074594580 shutdown_connections 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 --2- 172.21.15.146:0/4074594580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93680ccfb0 0x7f93680cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 --2- 172.21.15.146:0/4074594580 >> v2:172.21.15.177:3300/0 conn(0x7f936806cde0 0x7f936806d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 --2- 172.21.15.146:0/4074594580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368160e00 0x7f936815c490 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 -- 172.21.15.146:0/4074594580 >> 172.21.15.146:0/4074594580 conn(0x7f93680ce0c0 msgr2=0x7f93680ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 -- 172.21.15.146:0/4074594580 shutdown_connections 2024-04-25T15:22:56.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.916+0000 7f936ed34640 1 -- 172.21.15.146:0/4074594580 wait complete. 2024-04-25T15:22:56.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.918+0000 7f936ed34640 1 Processor -- start 2024-04-25T15:22:56.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.919+0000 7f936ed34640 1 -- start start 2024-04-25T15:22:56.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936ed34640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f936806cde0 0x7f9368150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936ed34640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 0x7f936809f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936ed34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f936815f3b0 0x7f93680b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936ed34640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9368160600 con 0x7f93680ccfb0 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936ed34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9368160320 con 0x7f936815f3b0 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936ed34640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f93681608e0 con 0x7f936806cde0 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936cad0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f936806cde0 0x7f9368150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 0x7f936809f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 0x7f936809f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58470/0 (socket says 172.21.15.146:58470) 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 -- 172.21.15.146:0/779313700 learned_addr learned my addr 172.21.15.146:0/779313700 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f936d2d1640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f936815f3b0 0x7f93680b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 -- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f936806cde0 msgr2=0x7f9368150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f936806cde0 0x7f9368150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 -- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f936815f3b0 msgr2=0x7f93680b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f936815f3b0 0x7f93680b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:56.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.921+0000 7f9367fff640 1 -- 172.21.15.146:0/779313700 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9368150db0 con 0x7f93680ccfb0 2024-04-25T15:22:56.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.922+0000 7f9367fff640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 0x7f936809f1d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f935400e870 tx=0x7f935400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:56.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.922+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354017070 con 0x7f93680ccfb0 2024-04-25T15:22:56.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.922+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f935400c940 con 0x7f93680ccfb0 2024-04-25T15:22:56.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.922+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9368151090 con 0x7f93680ccfb0 2024-04-25T15:22:56.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.922+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935400cd10 con 0x7f93680ccfb0 2024-04-25T15:22:56.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.922+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9368151650 con 0x7f93680ccfb0 2024-04-25T15:22:56.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.923+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f936806a0e0 con 0x7f93680ccfb0 2024-04-25T15:22:56.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.924+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f935401a070 con 0x7f93680ccfb0 2024-04-25T15:22:56.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.924+0000 7f93657fa640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f934c0745d0 0x7f934c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:56.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.924+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f93540957d0 con 0x7f93680ccfb0 2024-04-25T15:22:56.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.924+0000 7f936cad0640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f934c0745d0 0x7f934c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:56.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.925+0000 7f936cad0640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f934c0745d0 0x7f934c076ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f935c034000 tx=0x7f935c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:56.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:56.929+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f935405f5e0 con 0x7f93680ccfb0 2024-04-25T15:22:56.998 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/chfn' for reading: Permission denied 2024-04-25T15:22:57.015 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/chfn' for reading: Permission denied 2024-04-25T15:22:57.036 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/chfn' for reading: Permission denied 2024-04-25T15:22:57.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.199+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93680705d0 con 0x7f93680ccfb0 2024-04-25T15:22:57.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.200+0000 7f93657fa640 1 -- 172.21.15.146:0/779313700 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7f93540638e0 con 0x7f93680ccfb0 2024-04-25T15:22:57.201 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:22:57.202 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f934c0745d0 msgr2=0x7f934c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f934c0745d0 0x7f934c076ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f935c034000 tx=0x7f935c02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 msgr2=0x7f936809f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 0x7f936809f1d0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f935400e870 tx=0x7f935400ed30 comp rx=0 tx=0).stop 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 shutdown_connections 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f934c0745d0 0x7f934c076ae0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f936815f3b0 0x7f93680b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93680ccfb0 0x7f936809f1d0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 --2- 172.21.15.146:0/779313700 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f936806cde0 0x7f9368150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 >> 172.21.15.146:0/779313700 conn(0x7f93680ce0c0 msgr2=0x7f93680b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 shutdown_connections 2024-04-25T15:22:57.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.208+0000 7f936ed34640 1 -- 172.21.15.146:0/779313700 wait complete. 2024-04-25T15:22:57.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:22:57.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:22:57.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.361+0000 7fc5c3ad8640 1 Processor -- start 2024-04-25T15:22:57.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.361+0000 7fc5c3ad8640 1 -- start start 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.362+0000 7fc5c3ad8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5bc0b8f30 0x7fc5bc0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.362+0000 7fc5c3ad8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5bc08d590 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.362+0000 7fc5c3ad8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5bc0d27d0 con 0x7fc5bc08dd50 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.362+0000 7fc5c3ad8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc5bc0b9340 con 0x7fc5bc0b8f30 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c1073640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc5bc0b4ca0 0x7fc5bc0ce6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48216/0 (socket says 172.21.15.146:48216) 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c1874640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc5bc08dd50 0x7fc5bc0d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:35806/0 (socket says 172.21.15.146:35806) 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c1073640 1 -- 172.21.15.146:0/3042691191 learned_addr learned my addr 172.21.15.146:0/3042691191 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c2075640 1 --2- 172.21.15.146:0/3042691191 >> v2:172.21.15.146:3300/0 conn(0x7fc5bc0b8f30 0x7fc5bc0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126212500 0 0) 0x7fc5bc08d590 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a8003520 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983164598 0 0) 0x7fc5bc0d27d0 con 0x7fc5bc08dd50 2024-04-25T15:22:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5bc08d590 con 0x7fc5bc08dd50 2024-04-25T15:22:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c2075640 1 -- 172.21.15.146:0/3042691191 >> v1:172.21.15.177:6789/0 conn(0x7fc5bc08dd50 legacy=0x7fc5bc0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c2075640 1 -- 172.21.15.146:0/3042691191 >> v1:172.21.15.146:6789/0 conn(0x7fc5bc0b4ca0 legacy=0x7fc5bc0ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.363+0000 7fc5c2075640 1 -- 172.21.15.146:0/3042691191 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc0b48b0 con 0x7fc5bc0b8f30 2024-04-25T15:22:57.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.364+0000 7fc5c2075640 1 --2- 172.21.15.146:0/3042691191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b8f30 0x7fc5bc0cee00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009d90 tx=0x7fc5b402eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b87da9b3a248a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:57.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.364+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4030860 con 0x7fc5bc0b8f30 2024-04-25T15:22:57.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.364+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b4030a00 con 0x7fc5bc0b8f30 2024-04-25T15:22:57.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.364+0000 7fc5c0872640 1 -- 172.21.15.146:0/3042691191 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4030dd0 con 0x7fc5bc0b8f30 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/3042691191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b8f30 msgr2=0x7fc5bc0cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/3042691191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b8f30 0x7fc5bc0cee00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009d90 tx=0x7fc5b402eb00 comp rx=0 tx=0).stop 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/3042691191 shutdown_connections 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/3042691191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b8f30 0x7fc5bc0cee00 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/3042691191 >> 172.21.15.146:0/3042691191 conn(0x7fc5bc0b5430 msgr2=0x7fc5bc0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/3042691191 shutdown_connections 2024-04-25T15:22:57.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.365+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/3042691191 wait complete. 2024-04-25T15:22:57.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.368+0000 7fc5c3ad8640 1 Processor -- start 2024-04-25T15:22:57.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.368+0000 7fc5c3ad8640 1 -- start start 2024-04-25T15:22:57.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.369+0000 7fc5c3ad8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b4ca0 0x7fc5bc1f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.369+0000 7fc5c3ad8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 0x7fc5bc1d9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c3ad8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5bc1da7f0 0x7fc5bc0b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c3ad8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc0d2340 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c3ad8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc08d8c0 con 0x7fc5bc1da7f0 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c3ad8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc0d24c0 con 0x7fc5bc0b8f30 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1073640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 0x7fc5bc1d9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c2075640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5bc1da7f0 0x7fc5bc0b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1073640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 0x7fc5bc1d9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:45298/0 (socket says 172.21.15.146:45298) 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1073640 1 -- 172.21.15.146:0/2538137664 learned_addr learned my addr 172.21.15.146:0/2538137664 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1874640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b4ca0 0x7fc5bc1f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1874640 1 -- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 msgr2=0x7fc5bc1d9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1874640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 0x7fc5bc1d9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1874640 1 -- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5bc1da7f0 msgr2=0x7fc5bc0b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1874640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5bc1da7f0 0x7fc5bc0b79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1874640 1 -- 172.21.15.146:0/2538137664 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc0c7f90 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c1073640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 0x7fc5bc1d9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.370+0000 7fc5c2075640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5bc1da7f0 0x7fc5bc0b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.371+0000 7fc5c1874640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b4ca0 0x7fc5bc1f5b20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c3c0 tx=0x7fc5b8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.371+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80074e0 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.371+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b80076a0 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.371+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5bc0c8270 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.371+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8004a20 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.371+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5bc09f540 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.372+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5bc06a0e0 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.373+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fc5b801a020 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.374+0000 7fc5b27fc640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5a00745d0 0x7fc5a0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.374+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fc5b8094c00 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.374+0000 7fc5c1073640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5a00745d0 0x7fc5a0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.375+0000 7fc5c1073640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5a00745d0 0x7fc5a0076ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0cd4f0 tx=0x7fc5ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:57.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.380+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5b809b080 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.400 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/chsh' for reading: Permission denied 2024-04-25T15:22:57.401 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/chsh' for reading: Permission denied 2024-04-25T15:22:57.444 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/chsh' for reading: Permission denied 2024-04-25T15:22:57.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.615+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc5bc0d0d90 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.615+0000 7fc5b27fc640 1 -- 172.21.15.146:0/2538137664 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7fc5b805ea10 con 0x7fc5bc0b4ca0 2024-04-25T15:22:57.616 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5a00745d0 msgr2=0x7fc5a0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5a00745d0 0x7fc5a0076ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0cd4f0 tx=0x7fc5ac009040 comp rx=0 tx=0).stop 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b4ca0 msgr2=0x7fc5bc1f5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b4ca0 0x7fc5bc1f5b20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c3c0 tx=0x7fc5b8001e70 comp rx=0 tx=0).stop 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 shutdown_connections 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc5a00745d0 0x7fc5a0076ae0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5bc1da7f0 0x7fc5bc0b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5bc0b8f30 0x7fc5bc1d9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 --2- 172.21.15.146:0/2538137664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5bc0b4ca0 0x7fc5bc1f5b20 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 >> 172.21.15.146:0/2538137664 conn(0x7fc5bc0b5430 msgr2=0x7fc5bc0b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:57.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 shutdown_connections 2024-04-25T15:22:57.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.618+0000 7fc5c3ad8640 1 -- 172.21.15.146:0/2538137664 wait complete. 2024-04-25T15:22:57.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:22:57.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.767+0000 7f81af042640 1 Processor -- start 2024-04-25T15:22:57.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.767+0000 7f81af042640 1 -- start start 2024-04-25T15:22:57.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81af042640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f81a806e560 0x7f81a80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81af042640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a80cce80 con 0x7f81a81606b0 2024-04-25T15:22:57.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81af042640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a80cd160 con 0x7f81a80ceb70 2024-04-25T15:22:57.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81af042640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f81a80cd440 con 0x7f81a806e560 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81acdde640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f81a81606b0 0x7f81a80cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48226/0 (socket says 172.21.15.146:48226) 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81acdde640 1 -- 172.21.15.146:0/980835781 learned_addr learned my addr 172.21.15.146:0/980835781 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a7fff640 1 --2- 172.21.15.146:0/980835781 >> v2:172.21.15.177:3300/0 conn(0x7f81a806e560 0x7f81a80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 201445277 0 0) 0x7f81a80cd160 con 0x7f81a80ceb70 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8198003520 con 0x7f81a80ceb70 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760422784 0 0) 0x7f81a80cce80 con 0x7f81a81606b0 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81a80cd160 con 0x7f81a81606b0 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a7fff640 1 -- 172.21.15.146:0/980835781 >> v1:172.21.15.177:6789/0 conn(0x7f81a80ceb70 legacy=0x7f81a8166c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.769+0000 7f81a7fff640 1 -- 172.21.15.146:0/980835781 >> v1:172.21.15.146:6789/0 conn(0x7f81a81606b0 legacy=0x7f81a80cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.770+0000 7f81a7fff640 1 -- 172.21.15.146:0/980835781 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a80cd720 con 0x7f81a806e560 2024-04-25T15:22:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.770+0000 7f81a7fff640 1 --2- 172.21.15.146:0/980835781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a80ce630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f81940097a0 tx=0x7f819402c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a24d7d319179387b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:57.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.771+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194038070 con 0x7f81a806e560 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.771+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8194031710 con 0x7f81a806e560 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.771+0000 7f81a77fe640 1 -- 172.21.15.146:0/980835781 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194031ae0 con 0x7f81a806e560 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 -- 172.21.15.146:0/980835781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 msgr2=0x7f81a80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 --2- 172.21.15.146:0/980835781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a80ce630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f81940097a0 tx=0x7f819402c440 comp rx=0 tx=0).stop 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 -- 172.21.15.146:0/980835781 shutdown_connections 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 --2- 172.21.15.146:0/980835781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a80ce630 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 -- 172.21.15.146:0/980835781 >> 172.21.15.146:0/980835781 conn(0x7f81a809b2b0 msgr2=0x7f81a80b6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 -- 172.21.15.146:0/980835781 shutdown_connections 2024-04-25T15:22:57.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.772+0000 7f81af042640 1 -- 172.21.15.146:0/980835781 wait complete. 2024-04-25T15:22:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.774+0000 7f81af042640 1 Processor -- start 2024-04-25T15:22:57.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.775+0000 7f81af042640 1 -- start start 2024-04-25T15:22:57.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81af042640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a81fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81af042640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81a80ceb70 0x7f81a80cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81af042640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 0x7f81a80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81af042640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81a80ccb90 con 0x7f81a81606b0 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81af042640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81a80cca10 con 0x7f81a806e560 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81af042640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f81a80cef80 con 0x7f81a80ceb70 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81ad5df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 0x7f81a80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81a7fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81a80ceb70 0x7f81a80cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81ad5df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 0x7f81a80ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58494/0 (socket says 172.21.15.146:58494) 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81ad5df640 1 -- 172.21.15.146:0/1170154059 learned_addr learned my addr 172.21.15.146:0/1170154059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.776+0000 7f81acdde640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a81fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81ad5df640 1 -- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81a80ceb70 msgr2=0x7f81a80cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81ad5df640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81a80ceb70 0x7f81a80cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81ad5df640 1 -- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 msgr2=0x7f81a81fabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:57.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81ad5df640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a81fabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81ad5df640 1 -- 172.21.15.146:0/1170154059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a815da80 con 0x7f81a81606b0 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81acdde640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a81fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81ad5df640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 0x7f81a80ce360 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f81a000c6e0 tx=0x7f81a000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a0003b70 con 0x7f81a81606b0 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a0003d10 con 0x7f81a81606b0 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a81ee490 con 0x7f81a81606b0 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a0005310 con 0x7f81a81606b0 2024-04-25T15:22:57.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.777+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a81fb770 con 0x7f81a81606b0 2024-04-25T15:22:57.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.779+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a806a0e0 con 0x7f81a81606b0 2024-04-25T15:22:57.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.779+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f81a00054b0 con 0x7f81a81606b0 2024-04-25T15:22:57.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.780+0000 7f81a57fa640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f818c0745d0 0x7f818c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:57.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.780+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f81a0095300 con 0x7f81a81606b0 2024-04-25T15:22:57.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.780+0000 7f81acdde640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f818c0745d0 0x7f818c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:57.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.781+0000 7f81acdde640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f818c0745d0 0x7f818c076ae0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f819c000f60 tx=0x7f819c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:57.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:57.787+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81a005f110 con 0x7f81a81606b0 2024-04-25T15:22:57.790 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/locate' for reading: Permission denied 2024-04-25T15:22:57.790 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/bin/locate' for reading: Permission denied 2024-04-25T15:22:57.879 INFO:teuthology.orchestra.run.smithi146.stderr:cp: cannot open '/usr/bin/locate' for reading: Permission denied 2024-04-25T15:22:58.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.032+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81a81ee1b0 con 0x7f81a81606b0 2024-04-25T15:22:58.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.032+0000 7f81a57fa640 1 -- 172.21.15.146:0/1170154059 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f81a0063410 con 0x7f81a81606b0 2024-04-25T15:22:58.033 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f818c0745d0 msgr2=0x7f818c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f818c0745d0 0x7f818c076ae0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f819c000f60 tx=0x7f819c009040 comp rx=0 tx=0).stop 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 msgr2=0x7f81a80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 0x7f81a80ce360 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f81a000c6e0 tx=0x7f81a000cba0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 shutdown_connections 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f818c0745d0 0x7f818c076ae0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a81606b0 0x7f81a80ce360 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81a80ceb70 0x7f81a80cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 --2- 172.21.15.146:0/1170154059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81a806e560 0x7f81a81fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 >> 172.21.15.146:0/1170154059 conn(0x7f81a809b2b0 msgr2=0x7f81a80b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 shutdown_connections 2024-04-25T15:22:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.035+0000 7f81af042640 1 -- 172.21.15.146:0/1170154059 wait complete. 2024-04-25T15:22:58.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:22:58.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.184+0000 7f0116630640 1 Processor -- start 2024-04-25T15:22:58.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.184+0000 7f0116630640 1 -- start start 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f0116630640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01101602e0 0x7f01100b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f0116630640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01100d2480 con 0x7f011009b5e0 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f0116630640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01100d2740 con 0x7f01100cd290 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f0116630640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f01101606f0 con 0x7f01101602e0 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01101602e0 0x7f01100b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f011009b5e0 0x7f011009b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48240/0 (socket says 172.21.15.146:48240) 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010f7fe640 1 -- 172.21.15.146:0/160791292 learned_addr learned my addr 172.21.15.146:0/160791292 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010ffff640 1 -- 172.21.15.146:0/160791292 >> v1:172.21.15.177:6789/0 conn(0x7f01100cd290 legacy=0x7f01100cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010ffff640 1 -- 172.21.15.146:0/160791292 >> v1:172.21.15.146:6789/0 conn(0x7f011009b5e0 legacy=0x7f011009b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010effd640 1 -- 172.21.15.146:0/160791292 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217553889 0 0) 0x7f01100d2480 con 0x7f011009b5e0 2024-04-25T15:22:58.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.186+0000 7f010ffff640 1 -- 172.21.15.146:0/160791292 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01100d2a20 con 0x7f01101602e0 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.187+0000 7f010ffff640 1 --2- 172.21.15.146:0/160791292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01101602e0 0x7f01100b18c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0100009720 tx=0x7f010002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2acb3bca61418c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.187+0000 7f010effd640 1 -- 172.21.15.146:0/160791292 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100037070 con 0x7f01101602e0 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.187+0000 7f010effd640 1 -- 172.21.15.146:0/160791292 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f010002ade0 con 0x7f01101602e0 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.187+0000 7f010effd640 1 -- 172.21.15.146:0/160791292 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100032660 con 0x7f01101602e0 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 -- 172.21.15.146:0/160791292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01101602e0 msgr2=0x7f01100b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 --2- 172.21.15.146:0/160791292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01101602e0 0x7f01100b18c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0100009720 tx=0x7f010002a0d0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 -- 172.21.15.146:0/160791292 shutdown_connections 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 --2- 172.21.15.146:0/160791292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01101602e0 0x7f01100b18c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 -- 172.21.15.146:0/160791292 >> 172.21.15.146:0/160791292 conn(0x7f01100b7490 msgr2=0x7f01100b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:58.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 -- 172.21.15.146:0/160791292 shutdown_connections 2024-04-25T15:22:58.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.188+0000 7f0116630640 1 -- 172.21.15.146:0/160791292 wait complete. 2024-04-25T15:22:58.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.190+0000 7f0116630640 1 Processor -- start 2024-04-25T15:22:58.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.190+0000 7f0116630640 1 -- start start 2024-04-25T15:22:58.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.191+0000 7f0116630640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011009b5e0 0x7f01101f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.191+0000 7f0116630640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 0x7f01101f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.191+0000 7f0116630640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01101602e0 0x7f0110160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.191+0000 7f0116630640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f011006f7f0 con 0x7f01100cd290 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.191+0000 7f0116630640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01100d2fe0 con 0x7f011009b5e0 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.191+0000 7f0116630640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f01100b1e00 con 0x7f01101602e0 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 0x7f01101f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 0x7f01101f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58504/0 (socket says 172.21.15.146:58504) 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 -- 172.21.15.146:0/3096161293 learned_addr learned my addr 172.21.15.146:0/3096161293 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010ffff640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011009b5e0 0x7f01101f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 -- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01101602e0 msgr2=0x7f0110160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01101602e0 0x7f0110160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 -- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011009b5e0 msgr2=0x7f01101f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011009b5e0 0x7f01101f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 -- 172.21.15.146:0/3096161293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011014d830 con 0x7f01100cd290 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010f7fe640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 0x7f01101f5f60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f010400c470 tx=0x7f010400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:58.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01040074e0 con 0x7f01100cd290 2024-04-25T15:22:58.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01040047e0 con 0x7f01100cd290 2024-04-25T15:22:58.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01100c9120 con 0x7f01100cd290 2024-04-25T15:22:58.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104004bd0 con 0x7f01100cd290 2024-04-25T15:22:58.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.192+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01100cb360 con 0x7f01100cd290 2024-04-25T15:22:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.194+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f011006a0e0 con 0x7f01100cd290 2024-04-25T15:22:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.194+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f0104011070 con 0x7f01100cd290 2024-04-25T15:22:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.195+0000 7f010cff9640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00fc0745d0 0x7f00fc076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.195+0000 7f010ffff640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00fc0745d0 0x7f00fc076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.195+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f01040950a0 con 0x7f01100cd290 2024-04-25T15:22:58.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.196+0000 7f010ffff640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00fc0745d0 0x7f00fc076ae0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0100034000 tx=0x7f010002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:58.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.201+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f010405f620 con 0x7f01100cd290 2024-04-25T15:22:58.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.480+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f01100c8e40 con 0x7f01100cd290 2024-04-25T15:22:58.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.480+0000 7f010cff9640 1 -- 172.21.15.146:0/3096161293 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7f010405ef20 con 0x7f01100cd290 2024-04-25T15:22:58.481 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:22:58.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.482+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00fc0745d0 msgr2=0x7f00fc076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.482+0000 7f0116630640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00fc0745d0 0x7f00fc076ae0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0100034000 tx=0x7f010002e000 comp rx=0 tx=0).stop 2024-04-25T15:22:58.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.482+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 msgr2=0x7f01101f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.482+0000 7f0116630640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 0x7f01101f5f60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f010400c470 tx=0x7f010400c930 comp rx=0 tx=0).stop 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 shutdown_connections 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f00fc0745d0 0x7f00fc076ae0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01101602e0 0x7f0110160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01100cd290 0x7f01101f5f60 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 --2- 172.21.15.146:0/3096161293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f011009b5e0 0x7f01101f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 >> 172.21.15.146:0/3096161293 conn(0x7f01100b7490 msgr2=0x7f011015ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 shutdown_connections 2024-04-25T15:22:58.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.483+0000 7f0116630640 1 -- 172.21.15.146:0/3096161293 wait complete. 2024-04-25T15:22:58.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:22:58.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.635+0000 7fc0e7051640 1 Processor -- start 2024-04-25T15:22:58.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.635+0000 7fc0e7051640 1 -- start start 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e7051640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 0x7fc0e00d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e7051640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc0e006ccc0 0x7fc0e006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e7051640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e00b9200 con 0x7fc0e00d4f60 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e7051640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e006c550 con 0x7fc0e006ccc0 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e7051640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e006c830 con 0x7fc0e00d0730 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 0x7fc0e00d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 0x7fc0e00d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:45318/0 (socket says 172.21.15.146:45318) 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e55ee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc0e006ccc0 0x7fc0e006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0e4ded640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc0e00d4f60 0x7fc0e00d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:35836/0 (socket says 172.21.15.146:35836) 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.636+0000 7fc0dffff640 1 -- 172.21.15.146:0/2375635586 learned_addr learned my addr 172.21.15.146:0/2375635586 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.637+0000 7fc0df7fe640 1 -- 172.21.15.146:0/2375635586 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212843829 0 0) 0x7fc0d4001110 con 0x7fc0e00d4f60 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.637+0000 7fc0df7fe640 1 -- 172.21.15.146:0/2375635586 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0d0003520 con 0x7fc0e00d4f60 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.637+0000 7fc0dffff640 1 -- 172.21.15.146:0/2375635586 >> v2:172.21.15.177:3300/0 conn(0x7fc0e006ccc0 msgr2=0x7fc0e006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.637+0000 7fc0dffff640 1 --2- 172.21.15.146:0/2375635586 >> v2:172.21.15.177:3300/0 conn(0x7fc0e006ccc0 0x7fc0e006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.637+0000 7fc0dffff640 1 -- 172.21.15.146:0/2375635586 >> v1:172.21.15.177:6789/0 conn(0x7fc0e00d4f60 legacy=0x7fc0e00d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.637+0000 7fc0dffff640 1 -- 172.21.15.146:0/2375635586 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e00d1070 con 0x7fc0e00d0730 2024-04-25T15:22:58.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.638+0000 7fc0dffff640 1 --2- 172.21.15.146:0/2375635586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 0x7fc0e00d5370 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009870 tx=0x7fc0cc02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb826f7372bf1f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:58.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.638+0000 7fc0df7fe640 1 -- 172.21.15.146:0/2375635586 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc037070 con 0x7fc0e00d0730 2024-04-25T15:22:58.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.638+0000 7fc0df7fe640 1 -- 172.21.15.146:0/2375635586 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc032440 con 0x7fc0e00d0730 2024-04-25T15:22:58.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.638+0000 7fc0df7fe640 1 -- 172.21.15.146:0/2375635586 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc032810 con 0x7fc0e00d0730 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.639+0000 7fc0e7051640 1 -- 172.21.15.146:0/2375635586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 msgr2=0x7fc0e00d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.639+0000 7fc0e7051640 1 --2- 172.21.15.146:0/2375635586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 0x7fc0e00d5370 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009870 tx=0x7fc0cc02a130 comp rx=0 tx=0).stop 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.639+0000 7fc0e7051640 1 -- 172.21.15.146:0/2375635586 shutdown_connections 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.639+0000 7fc0e7051640 1 --2- 172.21.15.146:0/2375635586 >> v2:172.21.15.177:3300/0 conn(0x7fc0e006ccc0 0x7fc0e006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.639+0000 7fc0e7051640 1 --2- 172.21.15.146:0/2375635586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e00d0730 0x7fc0e00d5370 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.639+0000 7fc0e7051640 1 -- 172.21.15.146:0/2375635586 >> 172.21.15.146:0/2375635586 conn(0x7fc0e00d2450 msgr2=0x7fc0e00d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.640+0000 7fc0e7051640 1 -- 172.21.15.146:0/2375635586 shutdown_connections 2024-04-25T15:22:58.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.640+0000 7fc0e7051640 1 -- 172.21.15.146:0/2375635586 wait complete. 2024-04-25T15:22:58.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.642+0000 7fc0e7051640 1 Processor -- start 2024-04-25T15:22:58.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.642+0000 7fc0e7051640 1 -- start start 2024-04-25T15:22:58.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e7051640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 0x7fc0e00c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e7051640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0e00d4f60 0x7fc0e01fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e7051640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0e00b61e0 0x7fc0e00b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e7051640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e00b5150 con 0x7fc0e00d4f60 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e7051640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e00d15f0 con 0x7fc0e00b61e0 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e7051640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e00b52d0 con 0x7fc0e006ccc0 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0e00d4f60 0x7fc0e01fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e4ded640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 0x7fc0e00c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e4ded640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 0x7fc0e00c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:45330/0 (socket says 172.21.15.146:45330) 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e4ded640 1 -- 172.21.15.146:0/1715929760 learned_addr learned my addr 172.21.15.146:0/1715929760 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.644+0000 7fc0e55ee640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0e00b61e0 0x7fc0e00b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e4ded640 1 -- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0e00b61e0 msgr2=0x7fc0e00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e4ded640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0e00b61e0 0x7fc0e00b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e4ded640 1 -- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0e00d4f60 msgr2=0x7fc0e01fa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e4ded640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0e00d4f60 0x7fc0e01fa6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e4ded640 1 -- 172.21.15.146:0/1715929760 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e014b010 con 0x7fc0e006ccc0 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e55ee640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0e00b61e0 0x7fc0e00b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e4ded640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 0x7fc0e00c8e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400c3f0 tx=0x7fc0d400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4017070 con 0x7fc0e006ccc0 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4005c20 con 0x7fc0e006ccc0 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40070d0 con 0x7fc0e006ccc0 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e01fac20 con 0x7fc0e006ccc0 2024-04-25T15:22:58.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.645+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e014b500 con 0x7fc0e006ccc0 2024-04-25T15:22:58.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.647+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e006a0e0 con 0x7fc0e006ccc0 2024-04-25T15:22:58.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.647+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fc0d401a070 con 0x7fc0e006ccc0 2024-04-25T15:22:58.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.648+0000 7fc0dd7fa640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0c40745d0 0x7fc0c4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:22:58.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.648+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fc0d40948b0 con 0x7fc0e006ccc0 2024-04-25T15:22:58.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.648+0000 7fc0dffff640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0c40745d0 0x7fc0c4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:22:58.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.648+0000 7fc0dffff640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0c40745d0 0x7fc0c4076ae0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc034000 tx=0x7fc0cc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:22:58.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.654+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0e01fac20 con 0x7fc0e006ccc0 2024-04-25T15:22:58.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.938+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc0e00d4100 con 0x7fc0e006ccc0 2024-04-25T15:22:58.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.939+0000 7fc0dd7fa640 1 -- 172.21.15.146:0/1715929760 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v57) ==== 80+0+122 (secure 0 0 0) 0x7fc0e00b9200 con 0x7fc0e006ccc0 2024-04-25T15:22:58.940 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6836/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:22:58.940 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6834/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:22:58.940 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:22:58.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0c40745d0 msgr2=0x7fc0c4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0c40745d0 0x7fc0c4076ae0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc034000 tx=0x7fc0cc02e000 comp rx=0 tx=0).stop 2024-04-25T15:22:58.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 msgr2=0x7fc0e00c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:22:58.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 0x7fc0e00c8e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400c3f0 tx=0x7fc0d400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 shutdown_connections 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc0c40745d0 0x7fc0c4076ae0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0e00b61e0 0x7fc0e00b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0e00d4f60 0x7fc0e01fa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 --2- 172.21.15.146:0/1715929760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0e006ccc0 0x7fc0e00c8e20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.942+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 >> 172.21.15.146:0/1715929760 conn(0x7fc0e00d2450 msgr2=0x7fc0e015aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.943+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 shutdown_connections 2024-04-25T15:22:58.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:22:58.943+0000 7fc0e7051640 1 -- 172.21.15.146:0/1715929760 wait complete. 2024-04-25T15:22:58.956 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:22:58.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:58.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T15:22:58.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi146.stderr:ceph-fuse[59361]: fuse finished with error 0 and tester_r 0 2024-04-25T15:22:58.985 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:22:58.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:22:58.985 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:22:59.155 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:22:59.155 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--timeout=1', '--', 'bash', '-c', Raw("'\n set -ex\n pushd `mktemp -d -p .`\n cp -a /usr .\n popd\n '")], check_status=True, hostname='smithi177') 2024-04-25T15:22:59.157 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:22:59 stdin-killer INFO: expiration expected; waiting 1 seconds for command to complete 2024-04-25T15:23:00.156 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:23:00 stdin-killer ERROR: timeout expired: sending signal 9 to command and myself 2024-04-25T15:23:00.156 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 6: 44808 Killed ( cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:23:00.157 INFO:teuthology.orchestra.run.smithi177.stderr: set -ex 2024-04-25T15:23:00.157 INFO:teuthology.orchestra.run.smithi177.stderr: pushd `mktemp -d -p .` 2024-04-25T15:23:00.157 INFO:teuthology.orchestra.run.smithi177.stderr: cp -a /usr . 2024-04-25T15:23:00.157 INFO:teuthology.orchestra.run.smithi177.stderr: popd 2024-04-25T15:23:00.157 INFO:teuthology.orchestra.run.smithi177.stderr: ' ) 2024-04-25T15:23:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 137 2024-04-25T15:23:00.158 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:23:00.158 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:00.215 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:23:00.215 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:00.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:00.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.400+0000 7fc753f4d640 1 Processor -- start 2024-04-25T15:23:00.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.400+0000 7fc753f4d640 1 -- start start 2024-04-25T15:23:00.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc753f4d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc74c15def0 0x7fc74c1622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc753f4d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c0b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc753f4d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c0b5b90 con 0x7fc74c0ce020 2024-04-25T15:23:00.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc753f4d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc74c0d1950 con 0x7fc74c15def0 2024-04-25T15:23:00.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc753f4d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc74c0b7090 con 0x7fc74c0b5240 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc7514e8640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc74c0ce020 0x7fc74c0b6980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:48246/0 (socket says 172.21.15.146:48246) 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc7514e8640 1 -- 172.21.15.146:0/253228065 learned_addr learned my addr 172.21.15.146:0/253228065 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc7524ea640 1 --2- 172.21.15.146:0/253228065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c0b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc751ce9640 1 --2- 172.21.15.146:0/253228065 >> v2:172.21.15.177:3300/0 conn(0x7fc74c15def0 0x7fc74c1622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc750ce7640 1 -- 172.21.15.146:0/253228065 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246363743 0 0) 0x7fc73c001110 con 0x7fc74c0ce020 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc750ce7640 1 -- 172.21.15.146:0/253228065 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc738003520 con 0x7fc74c0ce020 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc7524ea640 1 -- 172.21.15.146:0/253228065 >> v2:172.21.15.177:3300/0 conn(0x7fc74c15def0 msgr2=0x7fc74c1622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc7524ea640 1 --2- 172.21.15.146:0/253228065 >> v2:172.21.15.177:3300/0 conn(0x7fc74c15def0 0x7fc74c1622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.402+0000 7fc7524ea640 1 -- 172.21.15.146:0/253228065 >> v1:172.21.15.146:6789/0 conn(0x7fc74c0ce020 legacy=0x7fc74c0b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.403+0000 7fc7524ea640 1 -- 172.21.15.146:0/253228065 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c0ce8c0 con 0x7fc74c0b5240 2024-04-25T15:23:00.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.403+0000 7fc751ce9640 1 --2- 172.21.15.146:0/253228065 >> v2:172.21.15.177:3300/0 conn(0x7fc74c15def0 0x7fc74c1622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:00.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.403+0000 7fc7524ea640 1 --2- 172.21.15.146:0/253228065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c0b5650 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc744009d90 tx=0x7fc74402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5d33768046af359 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:00.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.404+0000 7fc750ce7640 1 -- 172.21.15.146:0/253228065 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744007740 con 0x7fc74c0b5240 2024-04-25T15:23:00.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.404+0000 7fc750ce7640 1 -- 172.21.15.146:0/253228065 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7440078e0 con 0x7fc74c0b5240 2024-04-25T15:23:00.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.404+0000 7fc750ce7640 1 -- 172.21.15.146:0/253228065 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744007c90 con 0x7fc74c0b5240 2024-04-25T15:23:00.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 -- 172.21.15.146:0/253228065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 msgr2=0x7fc74c0b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 --2- 172.21.15.146:0/253228065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c0b5650 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc744009d90 tx=0x7fc74402a660 comp rx=0 tx=0).stop 2024-04-25T15:23:00.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 -- 172.21.15.146:0/253228065 shutdown_connections 2024-04-25T15:23:00.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 --2- 172.21.15.146:0/253228065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c0b5650 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 --2- 172.21.15.146:0/253228065 >> v2:172.21.15.177:3300/0 conn(0x7fc74c15def0 0x7fc74c1622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 -- 172.21.15.146:0/253228065 >> 172.21.15.146:0/253228065 conn(0x7fc74c0d0240 msgr2=0x7fc74c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:00.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 -- 172.21.15.146:0/253228065 shutdown_connections 2024-04-25T15:23:00.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.405+0000 7fc753f4d640 1 -- 172.21.15.146:0/253228065 wait complete. 2024-04-25T15:23:00.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.407+0000 7fc753f4d640 1 Processor -- start 2024-04-25T15:23:00.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.407+0000 7fc753f4d640 1 -- start start 2024-04-25T15:23:00.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.409+0000 7fc753f4d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c1f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.409+0000 7fc753f4d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c0ce020 0x7fc74c0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.409+0000 7fc753f4d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 0x7fc74c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.409+0000 7fc753f4d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c0d20c0 con 0x7fc74c15def0 2024-04-25T15:23:00.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.409+0000 7fc753f4d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c0cee80 con 0x7fc74c0ce020 2024-04-25T15:23:00.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.409+0000 7fc753f4d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc74c0d23a0 con 0x7fc74c0b5240 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 0x7fc74c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 0x7fc74c0b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58520/0 (socket says 172.21.15.146:58520) 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 -- 172.21.15.146:0/283337224 learned_addr learned my addr 172.21.15.146:0/283337224 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7514e8640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c0ce020 0x7fc74c0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc751ce9640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c1f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 -- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 msgr2=0x7fc74c1f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c1f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 -- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c0ce020 msgr2=0x7fc74c0b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c0ce020 0x7fc74c0b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 -- 172.21.15.146:0/283337224 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c0b3f10 con 0x7fc74c15def0 2024-04-25T15:23:00.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7524ea640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 0x7fc74c0b45e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc74402ac50 tx=0x7fc744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:00.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74403f070 con 0x7fc74c15def0 2024-04-25T15:23:00.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc744031c40 con 0x7fc74c15def0 2024-04-25T15:23:00.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c15e3e0 con 0x7fc74c15def0 2024-04-25T15:23:00.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.410+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744009450 con 0x7fc74c15def0 2024-04-25T15:23:00.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.411+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74c15e9a0 con 0x7fc74c15def0 2024-04-25T15:23:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.412+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c06a0e0 con 0x7fc74c15def0 2024-04-25T15:23:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.413+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fc74403b410 con 0x7fc74c15def0 2024-04-25T15:23:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.414+0000 7fc7427fc640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc7300745d0 0x7fc730076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.414+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fc7440b68d0 con 0x7fc74c15def0 2024-04-25T15:23:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.414+0000 7fc751ce9640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc7300745d0 0x7fc730076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.414+0000 7fc751ce9640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc7300745d0 0x7fc730076ae0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc748009560 tx=0x7fc748009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:00.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.420+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc744080660 con 0x7fc74c15def0 2024-04-25T15:23:00.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.662+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc74c0b3ca0 con 0x7fc74c15def0 2024-04-25T15:23:00.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.663+0000 7fc7427fc640 1 -- 172.21.15.146:0/283337224 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7fc744084960 con 0x7fc74c15def0 2024-04-25T15:23:00.664 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:00.664 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc7300745d0 msgr2=0x7fc730076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc7300745d0 0x7fc730076ae0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc748009560 tx=0x7fc748009190 comp rx=0 tx=0).stop 2024-04-25T15:23:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 msgr2=0x7fc74c0b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 0x7fc74c0b45e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc74402ac50 tx=0x7fc744004a40 comp rx=0 tx=0).stop 2024-04-25T15:23:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 shutdown_connections 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc7300745d0 0x7fc730076ae0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc74c15def0 0x7fc74c0b45e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74c0ce020 0x7fc74c0b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 --2- 172.21.15.146:0/283337224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74c0b5240 0x7fc74c1f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 >> 172.21.15.146:0/283337224 conn(0x7fc74c0d0240 msgr2=0x7fc74c0b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 shutdown_connections 2024-04-25T15:23:00.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.666+0000 7fc753f4d640 1 -- 172.21.15.146:0/283337224 wait complete. 2024-04-25T15:23:00.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:23:00.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:23:00.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.827+0000 7f611c8d8640 1 Processor -- start 2024-04-25T15:23:00.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.828+0000 7f611c8d8640 1 -- start start 2024-04-25T15:23:00.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f611c8d8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f61180b4ca0 0x7f61180b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f611c8d8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61180ce6f0 0x7f61180ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f611c8d8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611808d590 con 0x7f611808dd50 2024-04-25T15:23:00.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f611c8d8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f611808d8c0 con 0x7f61180ce6f0 2024-04-25T15:23:00.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f611c8d8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f61180d27d0 con 0x7f61180b4ca0 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6116d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61180ce6f0 0x7f61180ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6115d9b640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f61180b4ca0 0x7f61180b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6116d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61180ce6f0 0x7f61180ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37242/0 (socket says 172.21.15.146:37242) 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6116d9d640 1 -- 172.21.15.146:0/2692462858 learned_addr learned my addr 172.21.15.146:0/2692462858 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6115d9b640 1 -- 172.21.15.146:0/2692462858 >> v2:172.21.15.146:3300/0 conn(0x7f61180ce6f0 msgr2=0x7f61180ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6115d9b640 1 --2- 172.21.15.146:0/2692462858 >> v2:172.21.15.146:3300/0 conn(0x7f61180ce6f0 0x7f61180ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6115d9b640 1 -- 172.21.15.146:0/2692462858 >> v1:172.21.15.177:6789/0 conn(0x7f611808dd50 legacy=0x7f61180d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f611559a640 1 -- 172.21.15.146:0/2692462858 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 63321388 0 0) 0x7f611808d590 con 0x7f611808dd50 2024-04-25T15:23:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.830+0000 7f6115d9b640 1 -- 172.21.15.146:0/2692462858 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61180b48b0 con 0x7f61180b4ca0 2024-04-25T15:23:00.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.831+0000 7f6115d9b640 1 --2- 172.21.15.146:0/2692462858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180b8f30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f611808da40 tx=0x7f610c02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed83e9c5a8c5695b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:00.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.831+0000 7f611559a640 1 -- 172.21.15.146:0/2692462858 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c037070 con 0x7f61180b4ca0 2024-04-25T15:23:00.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.831+0000 7f611559a640 1 -- 172.21.15.146:0/2692462858 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f610c032440 con 0x7f61180b4ca0 2024-04-25T15:23:00.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.831+0000 7f611559a640 1 -- 172.21.15.146:0/2692462858 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c032810 con 0x7f61180b4ca0 2024-04-25T15:23:00.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 -- 172.21.15.146:0/2692462858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 msgr2=0x7f61180b8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 --2- 172.21.15.146:0/2692462858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180b8f30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f611808da40 tx=0x7f610c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 -- 172.21.15.146:0/2692462858 shutdown_connections 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 --2- 172.21.15.146:0/2692462858 >> v2:172.21.15.146:3300/0 conn(0x7f61180ce6f0 0x7f61180ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 --2- 172.21.15.146:0/2692462858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180b8f30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 -- 172.21.15.146:0/2692462858 >> 172.21.15.146:0/2692462858 conn(0x7f61180b5430 msgr2=0x7f61180b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 -- 172.21.15.146:0/2692462858 shutdown_connections 2024-04-25T15:23:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.832+0000 7f611c8d8640 1 -- 172.21.15.146:0/2692462858 wait complete. 2024-04-25T15:23:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.834+0000 7f611c8d8640 1 Processor -- start 2024-04-25T15:23:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.834+0000 7f611c8d8640 1 -- start start 2024-04-25T15:23:00.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611c8d8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611c8d8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61180ce6f0 0x7f61181dc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611c8d8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 0x7f61180c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611c8d8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61180d24c0 con 0x7f611809f3d0 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611c8d8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61180d2340 con 0x7f61180b4ca0 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611c8d8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f61180cf040 con 0x7f61180ce6f0 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f6116d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 0x7f61180c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f6115d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61180ce6f0 0x7f61181dc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39338/0 (socket says 172.21.15.146:39338) 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f6115d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61180ce6f0 0x7f61181dc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:52520/0 (socket says 172.21.15.146:52520) 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 -- 172.21.15.146:0/200214104 learned_addr learned my addr 172.21.15.146:0/200214104 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:00.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f6116d9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 0x7f61180c99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37246/0 (socket says 172.21.15.146:37246) 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 -- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61180ce6f0 msgr2=0x7f61181dc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61180ce6f0 0x7f61181dc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 -- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 msgr2=0x7f61180c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 0x7f61180c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f611659c640 1 -- 172.21.15.146:0/200214104 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611809fbb0 con 0x7f61180b4ca0 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.836+0000 7f6116d9d640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 0x7f61180c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:00.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.837+0000 7f611659c640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180c8a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f610800c3e0 tx=0x7f610800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:00.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.837+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108017070 con 0x7f61180b4ca0 2024-04-25T15:23:00.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.837+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61080035f0 con 0x7f61180b4ca0 2024-04-25T15:23:00.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.837+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f611809fe90 con 0x7f61180b4ca0 2024-04-25T15:23:00.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.837+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61080039c0 con 0x7f61180b4ca0 2024-04-25T15:23:00.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.837+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61180cd0a0 con 0x7f61180b4ca0 2024-04-25T15:23:00.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.838+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61180a03b0 con 0x7f61180b4ca0 2024-04-25T15:23:00.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.840+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f610801a070 con 0x7f61180b4ca0 2024-04-25T15:23:00.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.840+0000 7f6106ffd640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60f80745d0 0x7f60f8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:00.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.840+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f61080948f0 con 0x7f61180b4ca0 2024-04-25T15:23:00.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.840+0000 7f6115d9b640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60f80745d0 0x7f60f8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:00.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.841+0000 7f6115d9b640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60f80745d0 0x7f60f8076ae0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f610c034000 tx=0x7f610c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:00.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:00.846+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f610805e700 con 0x7f61180b4ca0 2024-04-25T15:23:01.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.122+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f611806a0e0 con 0x7f61180b4ca0 2024-04-25T15:23:01.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.122+0000 7f6106ffd640 1 -- 172.21.15.146:0/200214104 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7f6108062a00 con 0x7f61180b4ca0 2024-04-25T15:23:01.123 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:23:01.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60f80745d0 msgr2=0x7f60f8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60f80745d0 0x7f60f8076ae0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f610c034000 tx=0x7f610c02e000 comp rx=0 tx=0).stop 2024-04-25T15:23:01.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 msgr2=0x7f61180c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180c8a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f610800c3e0 tx=0x7f610800c8a0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 shutdown_connections 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f60f80745d0 0x7f60f8076ae0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f611809f3d0 0x7f61180c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61180ce6f0 0x7f61181dc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 --2- 172.21.15.146:0/200214104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180b4ca0 0x7f61180c8a20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 >> 172.21.15.146:0/200214104 conn(0x7f61180b5430 msgr2=0x7f61180b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 shutdown_connections 2024-04-25T15:23:01.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.125+0000 7f611c8d8640 1 -- 172.21.15.146:0/200214104 wait complete. 2024-04-25T15:23:01.141 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:23:01.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.291+0000 7f887aae7640 1 Processor -- start 2024-04-25T15:23:01.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.291+0000 7f887aae7640 1 -- start start 2024-04-25T15:23:01.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f887aae7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88741602e0 0x7f88740b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f887aae7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f88740cd290 0x7f88740cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f887aae7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88740d2480 con 0x7f887409b5e0 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f887aae7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f88741606f0 con 0x7f88741602e0 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f887aae7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f88740d2740 con 0x7f88740cd290 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f8878883640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88741602e0 0x7f88740b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f8873fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f887409b5e0 0x7f887409b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50828/0 (socket says 172.21.15.146:50828) 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.292+0000 7f8873fff640 1 -- 172.21.15.146:0/4131085859 learned_addr learned my addr 172.21.15.146:0/4131085859 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.293+0000 7f8879084640 1 --2- 172.21.15.146:0/4131085859 >> v2:172.21.15.177:3300/0 conn(0x7f88740cd290 0x7f88740cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.293+0000 7f8878883640 1 -- 172.21.15.146:0/4131085859 >> v2:172.21.15.177:3300/0 conn(0x7f88740cd290 msgr2=0x7f88740cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.293+0000 7f8878883640 1 --2- 172.21.15.146:0/4131085859 >> v2:172.21.15.177:3300/0 conn(0x7f88740cd290 0x7f88740cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.293+0000 7f8878883640 1 -- 172.21.15.146:0/4131085859 >> v1:172.21.15.146:6789/0 conn(0x7f887409b5e0 legacy=0x7f887409b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.293+0000 7f8878883640 1 -- 172.21.15.146:0/4131085859 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88740d2a20 con 0x7f88741602e0 2024-04-25T15:23:01.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.294+0000 7f8878883640 1 --2- 172.21.15.146:0/4131085859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88741602e0 0x7f88740b18c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f887406f500 tx=0x7f886802a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbe46f50353a5c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:01.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.294+0000 7f88737fe640 1 -- 172.21.15.146:0/4131085859 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868037070 con 0x7f88741602e0 2024-04-25T15:23:01.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.294+0000 7f88737fe640 1 -- 172.21.15.146:0/4131085859 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f886802ad30 con 0x7f88741602e0 2024-04-25T15:23:01.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.294+0000 7f88737fe640 1 -- 172.21.15.146:0/4131085859 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868032660 con 0x7f88741602e0 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 -- 172.21.15.146:0/4131085859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88741602e0 msgr2=0x7f88740b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 --2- 172.21.15.146:0/4131085859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88741602e0 0x7f88740b18c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f887406f500 tx=0x7f886802a0a0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 -- 172.21.15.146:0/4131085859 shutdown_connections 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 --2- 172.21.15.146:0/4131085859 >> v2:172.21.15.177:3300/0 conn(0x7f88740cd290 0x7f88740cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 --2- 172.21.15.146:0/4131085859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88741602e0 0x7f88740b18c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 -- 172.21.15.146:0/4131085859 >> 172.21.15.146:0/4131085859 conn(0x7f88740b7490 msgr2=0x7f88740b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 -- 172.21.15.146:0/4131085859 shutdown_connections 2024-04-25T15:23:01.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.296+0000 7f887aae7640 1 -- 172.21.15.146:0/4131085859 wait complete. 2024-04-25T15:23:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.299+0000 7f887aae7640 1 Processor -- start 2024-04-25T15:23:01.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.299+0000 7f887aae7640 1 -- start start 2024-04-25T15:23:01.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f887aae7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f887409b5e0 0x7f88740c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f887aae7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 0x7f88740c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f887aae7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88741602e0 0x7f88740c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f887aae7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f887406f7f0 con 0x7f88740cd290 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f8873fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 0x7f88740c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f8873fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 0x7f88740c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37264/0 (socket says 172.21.15.146:37264) 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f8873fff640 1 -- 172.21.15.146:0/3430749627 learned_addr learned my addr 172.21.15.146:0/3430749627 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f887aae7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88740d2fe0 con 0x7f887409b5e0 2024-04-25T15:23:01.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f88740b1e00 con 0x7f88741602e0 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.301+0000 7f8878883640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f887409b5e0 0x7f88740c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8873fff640 1 -- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88741602e0 msgr2=0x7f88740c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8879084640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88741602e0 0x7f88740c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8873fff640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88741602e0 0x7f88740c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8873fff640 1 -- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f887409b5e0 msgr2=0x7f88740c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8873fff640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f887409b5e0 0x7f88740c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8873fff640 1 -- 172.21.15.146:0/3430749627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8874161b80 con 0x7f88740cd290 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8878883640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f887409b5e0 0x7f88740c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8879084640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88741602e0 0x7f88740c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:01.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f8873fff640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 0x7f88740c88e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f886000c5a0 tx=0x7f886000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:01.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860017070 con 0x7f88740cd290 2024-04-25T15:23:01.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88600074e0 con 0x7f88740cd290 2024-04-25T15:23:01.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860004a00 con 0x7f88740cd290 2024-04-25T15:23:01.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.302+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8874161e60 con 0x7f88740cd290 2024-04-25T15:23:01.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.303+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8874162420 con 0x7f88740cd290 2024-04-25T15:23:01.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.304+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887406a0e0 con 0x7f88740cd290 2024-04-25T15:23:01.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.305+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f886001a070 con 0x7f88740cd290 2024-04-25T15:23:01.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.305+0000 7f88717fa640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f88580745d0 0x7f8858076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.305+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f8860094ac0 con 0x7f88740cd290 2024-04-25T15:23:01.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.305+0000 7f8878883640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f88580745d0 0x7f8858076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.306+0000 7f8878883640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f88580745d0 0x7f8858076ae0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8868034000 tx=0x7f886802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:01.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.312+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f886005ef20 con 0x7f88740cd290 2024-04-25T15:23:01.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.600+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f88740b7b90 con 0x7f88740cd290 2024-04-25T15:23:01.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.600+0000 7f88717fa640 1 -- 172.21.15.146:0/3430749627 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f886005e820 con 0x7f88740cd290 2024-04-25T15:23:01.601 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f88580745d0 msgr2=0x7f8858076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f88580745d0 0x7f8858076ae0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8868034000 tx=0x7f886802e000 comp rx=0 tx=0).stop 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 msgr2=0x7f88740c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 0x7f88740c88e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f886000c5a0 tx=0x7f886000ca60 comp rx=0 tx=0).stop 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 shutdown_connections 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f88580745d0 0x7f8858076ae0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88741602e0 0x7f88740c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88740cd290 0x7f88740c88e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 --2- 172.21.15.146:0/3430749627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f887409b5e0 0x7f88740c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 >> 172.21.15.146:0/3430749627 conn(0x7f88740b7490 msgr2=0x7f887406edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:01.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 shutdown_connections 2024-04-25T15:23:01.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.603+0000 7f887aae7640 1 -- 172.21.15.146:0/3430749627 wait complete. 2024-04-25T15:23:01.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:23:01.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.807+0000 7f74a0420640 1 Processor -- start 2024-04-25T15:23:01.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.807+0000 7f74a0420640 1 -- start start 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f74a0420640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f74980bb450 0x7f74980d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f74a0420640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980d0de0 0x7f74980b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f74a0420640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74980b5230 con 0x7f7498163c00 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f74a0420640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f749806c8c0 con 0x7f74980bb450 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f749e1bc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7498163c00 0x7f749806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50832/0 (socket says 172.21.15.146:50832) 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f749d9bb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f74980bb450 0x7f74980d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f749e1bc640 1 -- 172.21.15.146:0/64416283 learned_addr learned my addr 172.21.15.146:0/64416283 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f74a0420640 1 -- 172.21.15.146:0/64416283 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f749806cba0 con 0x7f74980d0de0 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.809+0000 7f749e9bd640 1 --2- 172.21.15.146:0/64416283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980d0de0 0x7f74980b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749d1ba640 1 -- 172.21.15.146:0/64416283 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 787155594 0 0) 0x7f74980b5230 con 0x7f7498163c00 2024-04-25T15:23:01.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749d1ba640 1 -- 172.21.15.146:0/64416283 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7484003520 con 0x7f7498163c00 2024-04-25T15:23:01.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749e9bd640 1 -- 172.21.15.146:0/64416283 >> v2:172.21.15.146:3300/0 conn(0x7f74980bb450 msgr2=0x7f74980d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:01.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749e9bd640 1 --2- 172.21.15.146:0/64416283 >> v2:172.21.15.146:3300/0 conn(0x7f74980bb450 0x7f74980d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749e9bd640 1 -- 172.21.15.146:0/64416283 >> v1:172.21.15.146:6789/0 conn(0x7f7498163c00 legacy=0x7f749806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749e9bd640 1 -- 172.21.15.146:0/64416283 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749806ce80 con 0x7f74980d0de0 2024-04-25T15:23:01.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.810+0000 7f749d9bb640 1 --2- 172.21.15.146:0/64416283 >> v2:172.21.15.146:3300/0 conn(0x7f74980bb450 0x7f74980d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.811+0000 7f749e9bd640 1 --2- 172.21.15.146:0/64416283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980d0de0 0x7f74980b8fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7490009dc0 tx=0x7f749002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66c2dd106b7509ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.811+0000 7f749d1ba640 1 -- 172.21.15.146:0/64416283 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490007740 con 0x7f74980d0de0 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.811+0000 7f749d1ba640 1 -- 172.21.15.146:0/64416283 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74900078e0 con 0x7f74980d0de0 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.811+0000 7f749d1ba640 1 -- 172.21.15.146:0/64416283 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490007cb0 con 0x7f74980d0de0 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 -- 172.21.15.146:0/64416283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980d0de0 msgr2=0x7f74980b8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 --2- 172.21.15.146:0/64416283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980d0de0 0x7f74980b8fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7490009dc0 tx=0x7f749002a5f0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 -- 172.21.15.146:0/64416283 shutdown_connections 2024-04-25T15:23:01.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 --2- 172.21.15.146:0/64416283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980d0de0 0x7f74980b8fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 --2- 172.21.15.146:0/64416283 >> v2:172.21.15.146:3300/0 conn(0x7f74980bb450 0x7f74980d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 -- 172.21.15.146:0/64416283 >> 172.21.15.146:0/64416283 conn(0x7f74980b7970 msgr2=0x7f74980b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 -- 172.21.15.146:0/64416283 shutdown_connections 2024-04-25T15:23:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.812+0000 7f74a0420640 1 -- 172.21.15.146:0/64416283 wait complete. 2024-04-25T15:23:01.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.815+0000 7f74a0420640 1 Processor -- start 2024-04-25T15:23:01.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.815+0000 7f74a0420640 1 -- start start 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f74a0420640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980bb450 0x7f74981521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f74a0420640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 0x7f7498154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f74a0420640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7498163c00 0x7f7498155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f74a0420640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f74980bb860 con 0x7f74980d0de0 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f74a0420640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f749806c550 con 0x7f7498163c00 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f74a0420640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f74980bb9e0 con 0x7f74980bb450 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 0x7f7498154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 0x7f7498154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37272/0 (socket says 172.21.15.146:37272) 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 -- 172.21.15.146:0/3253901854 learned_addr learned my addr 172.21.15.146:0/3253901854 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749e1bc640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980bb450 0x7f74981521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749e9bd640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7498163c00 0x7f7498155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 -- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980bb450 msgr2=0x7f74981521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980bb450 0x7f74981521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 -- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7498163c00 msgr2=0x7f7498155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7498163c00 0x7f7498155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.817+0000 7f749d9bb640 1 -- 172.21.15.146:0/3253901854 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7498152cd0 con 0x7f74980d0de0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.818+0000 7f749d9bb640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 0x7f7498154e30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f748800e930 tx=0x7f748800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.818+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488017070 con 0x7f74980d0de0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.818+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f748800ca00 con 0x7f74980d0de0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.818+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7498152fb0 con 0x7f74980d0de0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.818+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7498153570 con 0x7f74980d0de0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.818+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748800cdf0 con 0x7f74980d0de0 2024-04-25T15:23:01.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.819+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7480005290 con 0x7f74980d0de0 2024-04-25T15:23:01.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.820+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f748801a070 con 0x7f74980d0de0 2024-04-25T15:23:01.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.822+0000 7f748effd640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f747c0745d0 0x7f747c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:01.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.822+0000 7f749e1bc640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f747c0745d0 0x7f747c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:01.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.823+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f7488095420 con 0x7f74980d0de0 2024-04-25T15:23:01.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.824+0000 7f749e1bc640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f747c0745d0 0x7f747c076ae0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7494005880 tx=0x7f7494009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:01.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:01.827+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f748805f220 con 0x7f74980d0de0 2024-04-25T15:23:02.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.130+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7480005850 con 0x7f74980d0de0 2024-04-25T15:23:02.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.131+0000 7f748effd640 1 -- 172.21.15.146:0/3253901854 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7f7488063520 con 0x7f74980d0de0 2024-04-25T15:23:02.131 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:23:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f747c0745d0 msgr2=0x7f747c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f747c0745d0 0x7f747c076ae0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7494005880 tx=0x7f7494009040 comp rx=0 tx=0).stop 2024-04-25T15:23:02.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 msgr2=0x7f7498154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 0x7f7498154e30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f748800e930 tx=0x7f748800edf0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 shutdown_connections 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f747c0745d0 0x7f747c076ae0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7498163c00 0x7f7498155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74980d0de0 0x7f7498154e30 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 --2- 172.21.15.146:0/3253901854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74980bb450 0x7f74981521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 >> 172.21.15.146:0/3253901854 conn(0x7f74980b7970 msgr2=0x7f74980b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.134+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 shutdown_connections 2024-04-25T15:23:02.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.135+0000 7f74a0420640 1 -- 172.21.15.146:0/3253901854 wait complete. 2024-04-25T15:23:02.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:23:02.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.304+0000 7fcfe2184640 1 Processor -- start 2024-04-25T15:23:02.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.304+0000 7fcfe2184640 1 -- start start 2024-04-25T15:23:02.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.305+0000 7fcfe2184640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 0x7fcfdc0d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.305+0000 7fcfe2184640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcfdc0d0020 0x7fcfdc0d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.305+0000 7fcfe2184640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc0b9200 con 0x7fcfdc163d80 2024-04-25T15:23:02.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.305+0000 7fcfe2184640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fcfdc06c550 con 0x7fcfdc0d0020 2024-04-25T15:23:02.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.305+0000 7fcfe2184640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcfdc06c830 con 0x7fcfdc0d4f60 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdbfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcfdc163d80 0x7fcfdc164190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50836/0 (socket says 172.21.15.146:50836) 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdbfff640 1 -- 172.21.15.146:0/3682151796 learned_addr learned my addr 172.21.15.146:0/3682151796 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdaffd640 1 --2- 172.21.15.146:0/3682151796 >> v2:172.21.15.177:3300/0 conn(0x7fcfdc0d0020 0x7fcfdc0d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdb7fe640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 0x7fcfdc0d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1169756473 0 0) 0x7fcfd4001490 con 0x7fcfdc163d80 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfc4003520 con 0x7fcfdc163d80 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 55871049 0 0) 0x7fcfc4003520 con 0x7fcfdc163d80 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 msgr2=0x7fcfdc0d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 0x7fcfdc0d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 >> v2:172.21.15.177:3300/0 conn(0x7fcfdc0d0020 msgr2=0x7fcfdc0d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 --2- 172.21.15.146:0/3682151796 >> v2:172.21.15.177:3300/0 conn(0x7fcfdc0d0020 0x7fcfdc0d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc0d0970 con 0x7fcfdc163d80 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fcfd4001bc0 con 0x7fcfdc163d80 2024-04-25T15:23:02.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 >> v1:172.21.15.146:6789/0 conn(0x7fcfdc163d80 legacy=0x7fcfdc164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 0x7fcfc40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfc4003f30 0x7fcfc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfc4004d40 0x7fcfc4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc06c550 con 0x7fcfc4004d40 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc06c830 con 0x7fcfc4003f30 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcfc400bb50 con 0x7fcfdc0d4f60 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdbfff640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfc4004d40 0x7fcfc4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdb7fe640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 0x7fcfc40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.306+0000 7fcfdaffd640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfc4003f30 0x7fcfc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.307+0000 7fcfdbfff640 1 -- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 msgr2=0x7fcfc40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.307+0000 7fcfdbfff640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc0d4f60 0x7fcfc40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.307+0000 7fcfdbfff640 1 -- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfc4003f30 msgr2=0x7fcfc4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.307+0000 7fcfdbfff640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfc4003f30 0x7fcfc4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.307+0000 7fcfdbfff640 1 -- 172.21.15.146:0/3682151796 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc4003520 con 0x7fcfc4004d40 2024-04-25T15:23:02.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.307+0000 7fcfdbfff640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfc4004d40 0x7fcfc4005150 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40013d0 tx=0x7fcfc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad19142fdeaaa1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:02.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400a6b0 con 0x7fcfc4004d40 2024-04-25T15:23:02.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd402f070 con 0x7fcfc4004d40 2024-04-25T15:23:02.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfda7fc640 1 -- 172.21.15.146:0/3682151796 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd402be20 con 0x7fcfc4004d40 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfe2184640 1 -- 172.21.15.146:0/3682151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfc4004d40 msgr2=0x7fcfc4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfe2184640 1 --2- 172.21.15.146:0/3682151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfc4004d40 0x7fcfc4005150 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40013d0 tx=0x7fcfc4005830 comp rx=0 tx=0).stop 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfdbfff640 1 -- 172.21.15.146:0/3682151796 reap_dead start 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfe2184640 1 -- 172.21.15.146:0/3682151796 shutdown_connections 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.308+0000 7fcfe2184640 1 -- 172.21.15.146:0/3682151796 >> 172.21.15.146:0/3682151796 conn(0x7fcfdc0d2450 msgr2=0x7fcfdc0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.309+0000 7fcfe2184640 1 -- 172.21.15.146:0/3682151796 shutdown_connections 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.309+0000 7fcfe2184640 1 -- 172.21.15.146:0/3682151796 wait complete. 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.310+0000 7fcfe2184640 1 Processor -- start 2024-04-25T15:23:02.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.311+0000 7fcfe2184640 1 -- start start 2024-04-25T15:23:02.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfe2184640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfdc156fd0 0x7fcfdc1613a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfe2184640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 0x7fcfdc15f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfe2184640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc15f980 0x7fcfdc1fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfe2184640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0d15f0 con 0x7fcfdc156fd0 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfe2184640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0d1070 con 0x7fcfdc1618e0 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfe2184640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcfdc0b5150 con 0x7fcfdc15f980 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfdaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 0x7fcfdc15f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfdaffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 0x7fcfdc15f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39392/0 (socket says 172.21.15.146:39392) 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfdaffd640 1 -- 172.21.15.146:0/210271091 learned_addr learned my addr 172.21.15.146:0/210271091 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfdb7fe640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfdc156fd0 0x7fcfdc1613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.312+0000 7fcfdbfff640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc15f980 0x7fcfdc1fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdaffd640 1 -- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc15f980 msgr2=0x7fcfdc1fbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdaffd640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc15f980 0x7fcfdc1fbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdaffd640 1 -- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfdc156fd0 msgr2=0x7fcfdc1613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdaffd640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfdc156fd0 0x7fcfdc1613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdaffd640 1 -- 172.21.15.146:0/210271091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc15fd90 con 0x7fcfdc1618e0 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdb7fe640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfdc156fd0 0x7fcfdc1613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdbfff640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc15f980 0x7fcfdc1fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfdaffd640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 0x7fcfdc15f440 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00119c0 tx=0x7fcfd0011dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0019070 con 0x7fcfdc1618e0 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd0004240 con 0x7fcfdc1618e0 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000f820 con 0x7fcfdc1618e0 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.313+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc1605e0 con 0x7fcfdc1618e0 2024-04-25T15:23:02.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.314+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc160ba0 con 0x7fcfdc1618e0 2024-04-25T15:23:02.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.315+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc0b4b60 con 0x7fcfdc1618e0 2024-04-25T15:23:02.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.316+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fcfd001c070 con 0x7fcfdc1618e0 2024-04-25T15:23:02.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.316+0000 7fcfc3fff640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcfc80745d0 0x7fcfc8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:02.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.316+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fcfd0095c00 con 0x7fcfdc1618e0 2024-04-25T15:23:02.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.317+0000 7fcfdb7fe640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcfc80745d0 0x7fcfc8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:02.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.318+0000 7fcfdb7fe640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcfc80745d0 0x7fcfc8076ae0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc159880 tx=0x7fcfcc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:02.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.322+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfdc0b4b60 con 0x7fcfdc1618e0 2024-04-25T15:23:02.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.625+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcfdc06a0e0 con 0x7fcfdc1618e0 2024-04-25T15:23:02.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.625+0000 7fcfc3fff640 1 -- 172.21.15.146:0/210271091 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v57) ==== 80+0+122 (secure 0 0 0) 0x7fcfd009c080 con 0x7fcfdc1618e0 2024-04-25T15:23:02.626 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6836/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:23:02.626 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6834/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:23:02.627 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcfc80745d0 msgr2=0x7fcfc8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcfc80745d0 0x7fcfc8076ae0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc159880 tx=0x7fcfcc009190 comp rx=0 tx=0).stop 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 msgr2=0x7fcfdc15f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 0x7fcfdc15f440 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00119c0 tx=0x7fcfd0011dc0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 shutdown_connections 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcfc80745d0 0x7fcfc8076ae0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfdc15f980 0x7fcfdc1fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfdc1618e0 0x7fcfdc15f440 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 --2- 172.21.15.146:0/210271091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfdc156fd0 0x7fcfdc1613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:02.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.630+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 >> 172.21.15.146:0/210271091 conn(0x7fcfdc0d2450 msgr2=0x7fcfdc0b5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:02.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.631+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 shutdown_connections 2024-04-25T15:23:02.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02.631+0000 7fcfe2184640 1 -- 172.21.15.146:0/210271091 wait complete. 2024-04-25T15:23:02.649 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:23:02.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:02.649 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-25T15:23:02.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi177.stderr:ceph-fuse[44340]: fuse finished with error 0 and tester_r 0 2024-04-25T15:23:02.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:23:02.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:02.691 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:23:02.865 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:23:02.866 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--timeout=1', '--', 'bash', '-c', Raw("'\n set -ex\n pushd `mktemp -d -p .`\n cp -a /usr .\n popd\n '")], check_status=True, hostname='smithi146') 2024-04-25T15:23:02.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:02 stdin-killer INFO: expiration expected; waiting 1 seconds for command to complete 2024-04-25T15:23:03.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:03 stdin-killer ERROR: timeout expired: sending signal 9 to command and myself 2024-04-25T15:23:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 137 2024-04-25T15:23:03.867 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 6: 59964 Killed ( cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:23:03.867 INFO:teuthology.orchestra.run.smithi146.stderr: set -ex 2024-04-25T15:23:03.867 INFO:teuthology.orchestra.run.smithi146.stderr: pushd `mktemp -d -p .` 2024-04-25T15:23:03.868 INFO:teuthology.orchestra.run.smithi146.stderr: cp -a /usr . 2024-04-25T15:23:03.868 INFO:teuthology.orchestra.run.smithi146.stderr: popd 2024-04-25T15:23:03.868 INFO:teuthology.orchestra.run.smithi146.stderr: ' ) 2024-04-25T15:23:03.868 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:23:03.868 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:03.923 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:23:03.923 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:03.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:04.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.136+0000 7fcad2134640 1 Processor -- start 2024-04-25T15:23:04.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.136+0000 7fcad2134640 1 -- start start 2024-04-25T15:23:04.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.137+0000 7fcad2134640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc08d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.137+0000 7fcad2134640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcacc0b4df0 0x7fcacc161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.137+0000 7fcad2134640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcacc162520 con 0x7fcacc162110 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.137+0000 7fcad2134640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcacc0d23a0 con 0x7fcacc0b4df0 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.137+0000 7fcad2134640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcacc0d1790 con 0x7fcacc162890 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcacc0b4df0 0x7fcacc161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcacc0b4df0 0x7fcacc161bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37300/0 (socket says 172.21.15.146:37300) 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 -- 172.21.15.146:0/1362517452 learned_addr learned my addr 172.21.15.146:0/1362517452 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacb7fe640 1 --2- 172.21.15.146:0/1362517452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc08d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 -- 172.21.15.146:0/1362517452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 msgr2=0x7fcacc08d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 --2- 172.21.15.146:0/1362517452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc08d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 -- 172.21.15.146:0/1362517452 >> v1:172.21.15.177:6789/0 conn(0x7fcacc162110 legacy=0x7fcacc15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.138+0000 7fcacaffd640 1 -- 172.21.15.146:0/1362517452 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcacc0d1a70 con 0x7fcacc0b4df0 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcacaffd640 1 --2- 172.21.15.146:0/1362517452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc161bd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009870 tx=0x7fcac002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d29edf56321965 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcaca7fc640 1 -- 172.21.15.146:0/1362517452 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0037070 con 0x7fcacc0b4df0 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcaca7fc640 1 -- 172.21.15.146:0/1362517452 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac0032440 con 0x7fcacc0b4df0 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcaca7fc640 1 -- 172.21.15.146:0/1362517452 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0032810 con 0x7fcacc0b4df0 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcad2134640 1 -- 172.21.15.146:0/1362517452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 msgr2=0x7fcacc161bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcad2134640 1 --2- 172.21.15.146:0/1362517452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc161bd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009870 tx=0x7fcac002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcad2134640 1 -- 172.21.15.146:0/1362517452 shutdown_connections 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcad2134640 1 --2- 172.21.15.146:0/1362517452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc161bd0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcad2134640 1 --2- 172.21.15.146:0/1362517452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc08d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.139+0000 7fcad2134640 1 -- 172.21.15.146:0/1362517452 >> 172.21.15.146:0/1362517452 conn(0x7fcacc0b52e0 msgr2=0x7fcacc0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.140+0000 7fcad2134640 1 -- 172.21.15.146:0/1362517452 shutdown_connections 2024-04-25T15:23:04.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.140+0000 7fcad2134640 1 -- 172.21.15.146:0/1362517452 wait complete. 2024-04-25T15:23:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.141+0000 7fcad2134640 1 Processor -- start 2024-04-25T15:23:04.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.141+0000 7fcad2134640 1 -- start start 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcad2134640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcad2134640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc162110 0x7fcacc152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcad2134640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc15fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcad2134640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc0d2830 con 0x7fcacc0b4df0 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcad2134640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc0d2030 con 0x7fcacc162110 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcad2134640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcacc08dbb0 con 0x7fcacc162890 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc0cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37314/0 (socket says 172.21.15.146:37314) 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 -- 172.21.15.146:0/3728764367 learned_addr learned my addr 172.21.15.146:0/3728764367 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacaffd640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc162110 0x7fcacc152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacbfff640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc15fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 -- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 msgr2=0x7fcacc15fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc15fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 -- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc162110 msgr2=0x7fcacc152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc162110 0x7fcacc152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.143+0000 7fcacb7fe640 1 -- 172.21.15.146:0/3728764367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcacc06e3d0 con 0x7fcacc0b4df0 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.144+0000 7fcacb7fe640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc0cc620 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00e870 tx=0x7fcabc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.144+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcabc017070 con 0x7fcacc0b4df0 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.144+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcabc00c940 con 0x7fcacc0b4df0 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.144+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcabc00ccf0 con 0x7fcacc0b4df0 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.144+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcacc153e20 con 0x7fcacc0b4df0 2024-04-25T15:23:04.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.144+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcacc06e920 con 0x7fcacc0b4df0 2024-04-25T15:23:04.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.146+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcacc06a0e0 con 0x7fcacc0b4df0 2024-04-25T15:23:04.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.146+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fcabc01a070 con 0x7fcacc0b4df0 2024-04-25T15:23:04.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.146+0000 7fcab3fff640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcab80745d0 0x7fcab8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.146+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fcabc095850 con 0x7fcacc0b4df0 2024-04-25T15:23:04.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.146+0000 7fcacaffd640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcab80745d0 0x7fcab8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.147+0000 7fcacaffd640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcab80745d0 0x7fcab8076ae0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcac0034000 tx=0x7fcac002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.154+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcabc05f720 con 0x7fcacc0b4df0 2024-04-25T15:23:04.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.383+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcacc0b61f0 con 0x7fcacc0b4df0 2024-04-25T15:23:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.384+0000 7fcab3fff640 1 -- 172.21.15.146:0/3728764367 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7fcabc063a20 con 0x7fcacc0b4df0 2024-04-25T15:23:04.385 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:04.385 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcab80745d0 msgr2=0x7fcab8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcab80745d0 0x7fcab8076ae0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcac0034000 tx=0x7fcac002e000 comp rx=0 tx=0).stop 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 msgr2=0x7fcacc0cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc0cc620 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00e870 tx=0x7fcabc00ed30 comp rx=0 tx=0).stop 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 shutdown_connections 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcab80745d0 0x7fcab8076ae0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcacc162890 0x7fcacc15fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc162110 0x7fcacc152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 --2- 172.21.15.146:0/3728764367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcacc0b4df0 0x7fcacc0cc620 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.386+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 >> 172.21.15.146:0/3728764367 conn(0x7fcacc0b52e0 msgr2=0x7fcacc0b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:04.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.387+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 shutdown_connections 2024-04-25T15:23:04.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.387+0000 7fcad2134640 1 -- 172.21.15.146:0/3728764367 wait complete. 2024-04-25T15:23:04.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:23:04.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:23:04.419 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/sbin/unix_update' for reading: Permission denied 2024-04-25T15:23:04.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.530+0000 7f548a70d640 1 Processor -- start 2024-04-25T15:23:04.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.530+0000 7f548a70d640 1 -- start start 2024-04-25T15:23:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.531+0000 7f548a70d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5484163c00 0x7f548406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.531+0000 7f548a70d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54840bb450 0x7f54840d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.531+0000 7f548a70d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f548406dea0 con 0x7f54840d0de0 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.531+0000 7f548a70d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f54840b5230 con 0x7f54840bb450 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.531+0000 7f548a70d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f548406c8c0 con 0x7f5484163c00 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54840bb450 0x7f54840d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54840bb450 0x7f54840d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37320/0 (socket says 172.21.15.146:37320) 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 -- 172.21.15.146:0/4169631018 learned_addr learned my addr 172.21.15.146:0/4169631018 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f5483fff640 1 --2- 172.21.15.146:0/4169631018 >> v2:172.21.15.177:3300/0 conn(0x7f5484163c00 0x7f548406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 -- 172.21.15.146:0/4169631018 >> v2:172.21.15.177:3300/0 conn(0x7f5484163c00 msgr2=0x7f548406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 --2- 172.21.15.146:0/4169631018 >> v2:172.21.15.177:3300/0 conn(0x7f5484163c00 0x7f548406d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 -- 172.21.15.146:0/4169631018 >> v1:172.21.15.177:6789/0 conn(0x7f54840d0de0 legacy=0x7f54840b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.532+0000 7f54837fe640 1 -- 172.21.15.146:0/4169631018 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f548406cba0 con 0x7f54840bb450 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.533+0000 7f54837fe640 1 --2- 172.21.15.146:0/4169631018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840bb450 0x7f54840d08a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f548408e920 tx=0x7f547802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443bf633e852d1f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.533+0000 7f5482ffd640 1 -- 172.21.15.146:0/4169631018 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478037070 con 0x7f54840bb450 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.533+0000 7f5482ffd640 1 -- 172.21.15.146:0/4169631018 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5478032440 con 0x7f54840bb450 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.533+0000 7f5482ffd640 1 -- 172.21.15.146:0/4169631018 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478032810 con 0x7f54840bb450 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 -- 172.21.15.146:0/4169631018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840bb450 msgr2=0x7f54840d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 --2- 172.21.15.146:0/4169631018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840bb450 0x7f54840d08a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f548408e920 tx=0x7f547802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 -- 172.21.15.146:0/4169631018 shutdown_connections 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 --2- 172.21.15.146:0/4169631018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840bb450 0x7f54840d08a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 --2- 172.21.15.146:0/4169631018 >> v2:172.21.15.177:3300/0 conn(0x7f5484163c00 0x7f548406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 -- 172.21.15.146:0/4169631018 >> 172.21.15.146:0/4169631018 conn(0x7f54840b7970 msgr2=0x7f54840b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 -- 172.21.15.146:0/4169631018 shutdown_connections 2024-04-25T15:23:04.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.534+0000 7f548a70d640 1 -- 172.21.15.146:0/4169631018 wait complete. 2024-04-25T15:23:04.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.536+0000 7f548a70d640 1 Processor -- start 2024-04-25T15:23:04.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.536+0000 7f548a70d640 1 -- start start 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f548a70d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54840bb450 0x7f548414ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f548a70d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 0x7f54840cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f548a70d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484163c00 0x7f54840cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f548a70d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f548406c550 con 0x7f54840d0de0 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f548a70d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f548406d160 con 0x7f5484163c00 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f548a70d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f54840bb860 con 0x7f54840bb450 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f54837fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 0x7f54840cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f54837fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 0x7f54840cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37324/0 (socket says 172.21.15.146:37324) 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f54837fe640 1 -- 172.21.15.146:0/3495053739 learned_addr learned my addr 172.21.15.146:0/3495053739 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f5483fff640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54840bb450 0x7f548414ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.537+0000 7f5488caa640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484163c00 0x7f54840cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f54837fe640 1 -- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54840bb450 msgr2=0x7f548414ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f54837fe640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54840bb450 0x7f548414ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f54837fe640 1 -- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484163c00 msgr2=0x7f54840cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f54837fe640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484163c00 0x7f54840cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f54837fe640 1 -- 172.21.15.146:0/3495053739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54840cc050 con 0x7f54840d0de0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f54837fe640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 0x7f54840cb290 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5478000c00 tx=0x7f54780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478037040 con 0x7f54840d0de0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5478042070 con 0x7f54840d0de0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548414f340 con 0x7f54840d0de0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478030880 con 0x7f54840d0de0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.538+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f548414f900 con 0x7f54840d0de0 2024-04-25T15:23:04.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.539+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f548406a0e0 con 0x7f54840d0de0 2024-04-25T15:23:04.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.540+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f5478030a20 con 0x7f54840d0de0 2024-04-25T15:23:04.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.540+0000 7f5480ff9640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5470074650 0x7f5470076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.540+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f54780b5ce0 con 0x7f54840d0de0 2024-04-25T15:23:04.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.540+0000 7f5483fff640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5470074650 0x7f5470076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.541+0000 7f5483fff640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5470074650 0x7f5470076b60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f54840b5130 tx=0x7f5474009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.545+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f547807fb60 con 0x7f54840d0de0 2024-04-25T15:23:04.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.759+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54840cf030 con 0x7f54840d0de0 2024-04-25T15:23:04.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.759+0000 7f5480ff9640 1 -- 172.21.15.146:0/3495053739 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7f54780881a0 con 0x7f54840d0de0 2024-04-25T15:23:04.760 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5470074650 msgr2=0x7f5470076b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5470074650 0x7f5470076b60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f54840b5130 tx=0x7f5474009190 comp rx=0 tx=0).stop 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 msgr2=0x7f54840cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 0x7f54840cb290 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5478000c00 tx=0x7f54780046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 shutdown_connections 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5470074650 0x7f5470076b60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5484163c00 0x7f54840cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54840d0de0 0x7f54840cb290 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 --2- 172.21.15.146:0/3495053739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54840bb450 0x7f548414ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 >> 172.21.15.146:0/3495053739 conn(0x7f54840b7970 msgr2=0x7f54840b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 shutdown_connections 2024-04-25T15:23:04.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.762+0000 7f548a70d640 1 -- 172.21.15.146:0/3495053739 wait complete. 2024-04-25T15:23:04.773 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:23:04.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.891+0000 7f1db46f9640 1 Processor -- start 2024-04-25T15:23:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.891+0000 7f1db46f9640 1 -- start start 2024-04-25T15:23:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db46f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0b2cd0 0x7f1dac0d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db46f9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1dac0d3300 0x7f1dac071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db46f9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac071f70 con 0x7f1dac071b60 2024-04-25T15:23:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db46f9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1dac0d2260 con 0x7f1dac0d3300 2024-04-25T15:23:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db46f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1dac0d2540 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1dac0d3300 0x7f1dac071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1dac0d3300 0x7f1dac071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37326/0 (socket says 172.21.15.146:37326) 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 -- 172.21.15.146:0/1087679343 learned_addr learned my addr 172.21.15.146:0/1087679343 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db2495640 1 --2- 172.21.15.146:0/1087679343 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0b2cd0 0x7f1dac0d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 -- 172.21.15.146:0/1087679343 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0b2cd0 msgr2=0x7f1dac0d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 --2- 172.21.15.146:0/1087679343 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0b2cd0 0x7f1dac0d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 -- 172.21.15.146:0/1087679343 >> v1:172.21.15.177:6789/0 conn(0x7f1dac071b60 legacy=0x7f1dac08cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.893+0000 7f1db1c94640 1 -- 172.21.15.146:0/1087679343 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac0d2820 con 0x7f1dac0d3300 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.894+0000 7f1db1c94640 1 --2- 172.21.15.146:0/1087679343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0d3300 0x7f1dac071620 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009870 tx=0x7f1d9c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ed2c4202364d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.894+0000 7f1db1493640 1 -- 172.21.15.146:0/1087679343 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c037070 con 0x7f1dac0d3300 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.894+0000 7f1db1493640 1 -- 172.21.15.146:0/1087679343 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c032440 con 0x7f1dac0d3300 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.894+0000 7f1db1493640 1 -- 172.21.15.146:0/1087679343 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c032810 con 0x7f1dac0d3300 2024-04-25T15:23:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 -- 172.21.15.146:0/1087679343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0d3300 msgr2=0x7f1dac071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1087679343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0d3300 0x7f1dac071620 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009870 tx=0x7f1d9c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 -- 172.21.15.146:0/1087679343 shutdown_connections 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1087679343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0d3300 0x7f1dac071620 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1087679343 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0b2cd0 0x7f1dac0d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 -- 172.21.15.146:0/1087679343 >> 172.21.15.146:0/1087679343 conn(0x7f1dac0b52e0 msgr2=0x7f1dac0b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 -- 172.21.15.146:0/1087679343 shutdown_connections 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.895+0000 7f1db46f9640 1 -- 172.21.15.146:0/1087679343 wait complete. 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.897+0000 7f1db46f9640 1 Processor -- start 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.897+0000 7f1db46f9640 1 -- start start 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.898+0000 7f1db46f9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1dac071b60 0x7f1dac1f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.898+0000 7f1db46f9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 0x7f1dac0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.898+0000 7f1db46f9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0d3300 0x7f1dac0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.898+0000 7f1db46f9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac0cfb70 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.898+0000 7f1db46f9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac0d2de0 con 0x7f1dac071b60 2024-04-25T15:23:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.898+0000 7f1db46f9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1dac08dbf0 con 0x7f1dac0d3300 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 0x7f1dac0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 0x7f1dac0c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37334/0 (socket says 172.21.15.146:37334) 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 -- 172.21.15.146:0/1210729415 learned_addr learned my addr 172.21.15.146:0/1210729415 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db2495640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1dac071b60 0x7f1dac1f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db2c96640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0d3300 0x7f1dac0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 -- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0d3300 msgr2=0x7f1dac0c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0d3300 0x7f1dac0c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 -- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1dac071b60 msgr2=0x7f1dac1f5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1dac071b60 0x7f1dac1f5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 -- 172.21.15.146:0/1210729415 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac0c7ea0 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.899+0000 7f1db1c94640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 0x7f1dac0c71b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c02a200 tx=0x7f1d9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.900+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c041030 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.900+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c030440 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.900+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c037040 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.900+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac0c8180 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.900+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dac0c8740 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.901+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dac06a0e0 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.901+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f1d9c044070 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.902+0000 7f1da2ffd640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1d90074600 0x7f1d90076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.902+0000 7f1db2495640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1d90074600 0x7f1d90076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.902+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f1d9c0b5970 con 0x7f1dac0b2cd0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.902+0000 7f1db2495640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1d90074600 0x7f1d90076b10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0b9490 tx=0x7f1da8009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:04.907+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d9c04b070 con 0x7f1dac0b2cd0 2024-04-25T15:23:05.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.107+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1dac0ced50 con 0x7f1dac0b2cd0 2024-04-25T15:23:05.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.107+0000 7f1da2ffd640 1 -- 172.21.15.146:0/1210729415 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f1d9c07f780 con 0x7f1dac0b2cd0 2024-04-25T15:23:05.108 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1d90074600 msgr2=0x7f1d90076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1d90074600 0x7f1d90076b10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1dac0b9490 tx=0x7f1da8009110 comp rx=0 tx=0).stop 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 msgr2=0x7f1dac0c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 0x7f1dac0c71b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c02a200 tx=0x7f1d9c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 shutdown_connections 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1d90074600 0x7f1d90076b10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1dac0d3300 0x7f1dac0c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dac0b2cd0 0x7f1dac0c71b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 --2- 172.21.15.146:0/1210729415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1dac071b60 0x7f1dac1f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 >> 172.21.15.146:0/1210729415 conn(0x7f1dac0b52e0 msgr2=0x7f1dac0cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:05.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 shutdown_connections 2024-04-25T15:23:05.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.110+0000 7f1db46f9640 1 -- 172.21.15.146:0/1210729415 wait complete. 2024-04-25T15:23:05.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:23:05.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.249+0000 7f11700e0640 1 Processor -- start 2024-04-25T15:23:05.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.249+0000 7f11700e0640 1 -- start start 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f11700e0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f11680b4ca0 0x7f11680b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f11700e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116808d590 con 0x7f11680ce6f0 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f11700e0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11680d27d0 con 0x7f116808dd50 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f11700e0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f116808d8c0 con 0x7f11680b4ca0 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116e67d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f11680ce6f0 0x7f11680ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50852/0 (socket says 172.21.15.146:50852) 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116e67d640 1 -- 172.21.15.146:0/607248367 learned_addr learned my addr 172.21.15.146:0/607248367 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116d67b640 1 --2- 172.21.15.146:0/607248367 >> v2:172.21.15.177:3300/0 conn(0x7f11680b4ca0 0x7f11680b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116ce7a640 1 -- 172.21.15.146:0/607248367 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411276072 0 0) 0x7f116808d590 con 0x7f11680ce6f0 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116ce7a640 1 -- 172.21.15.146:0/607248367 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1154003520 con 0x7f11680ce6f0 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116d67b640 1 -- 172.21.15.146:0/607248367 >> v1:172.21.15.177:6789/0 conn(0x7f116808dd50 legacy=0x7f11680d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116d67b640 1 -- 172.21.15.146:0/607248367 >> v1:172.21.15.146:6789/0 conn(0x7f11680ce6f0 legacy=0x7f11680ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.251+0000 7f116d67b640 1 -- 172.21.15.146:0/607248367 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11680b48b0 con 0x7f11680b4ca0 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.252+0000 7f116d67b640 1 --2- 172.21.15.146:0/607248367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680b4ca0 0x7f11680b8f30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f11580097a0 tx=0x7f115802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c20abdc6163a84c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.252+0000 7f116ce7a640 1 -- 172.21.15.146:0/607248367 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158037070 con 0x7f11680b4ca0 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.252+0000 7f116ce7a640 1 -- 172.21.15.146:0/607248367 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1158032440 con 0x7f11680b4ca0 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.252+0000 7f116ce7a640 1 -- 172.21.15.146:0/607248367 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11580327f0 con 0x7f11680b4ca0 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 -- 172.21.15.146:0/607248367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680b4ca0 msgr2=0x7f11680b8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 --2- 172.21.15.146:0/607248367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680b4ca0 0x7f11680b8f30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f11580097a0 tx=0x7f115802a230 comp rx=0 tx=0).stop 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 -- 172.21.15.146:0/607248367 shutdown_connections 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 --2- 172.21.15.146:0/607248367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680b4ca0 0x7f11680b8f30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 -- 172.21.15.146:0/607248367 >> 172.21.15.146:0/607248367 conn(0x7f11680b5430 msgr2=0x7f11680b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 -- 172.21.15.146:0/607248367 shutdown_connections 2024-04-25T15:23:05.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.253+0000 7f11700e0640 1 -- 172.21.15.146:0/607248367 wait complete. 2024-04-25T15:23:05.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.255+0000 7f11700e0640 1 Processor -- start 2024-04-25T15:23:05.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.255+0000 7f11700e0640 1 -- start start 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f11700e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 0x7f11681d8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f11700e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680ce6f0 0x7f11681f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f11700e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11681d8ca0 0x7f11680b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f11700e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f11680d24c0 con 0x7f11680b4ca0 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f11700e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f11680d2340 con 0x7f11680ce6f0 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f11700e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f11680cf210 con 0x7f11681d8ca0 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 0x7f11681d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 0x7f11681d8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37336/0 (socket says 172.21.15.146:37336) 2024-04-25T15:23:05.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 -- 172.21.15.146:0/3051506138 learned_addr learned my addr 172.21.15.146:0/3051506138 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116d67b640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680ce6f0 0x7f11681f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116e67d640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11681d8ca0 0x7f11680b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 -- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11681d8ca0 msgr2=0x7f11680b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11681d8ca0 0x7f11680b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 -- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680ce6f0 msgr2=0x7f11681f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680ce6f0 0x7f11681f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.256+0000 7f116de7c640 1 -- 172.21.15.146:0/3051506138 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11681f8160 con 0x7f11680b4ca0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.257+0000 7f116de7c640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 0x7f11681d8760 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f116400c3c0 tx=0x7f116400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.257+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164017070 con 0x7f11680b4ca0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.257+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11640052f0 con 0x7f11680b4ca0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.257+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f116809e030 con 0x7f11680b4ca0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.257+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164003830 con 0x7f11680b4ca0 2024-04-25T15:23:05.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.257+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f116809fb90 con 0x7f11680b4ca0 2024-04-25T15:23:05.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.258+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f116806a0e0 con 0x7f11680b4ca0 2024-04-25T15:23:05.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.259+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f116401a070 con 0x7f11680b4ca0 2024-04-25T15:23:05.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.260+0000 7f115e7fc640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f114c074600 0x7f114c076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.260+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f1164094780 con 0x7f11680b4ca0 2024-04-25T15:23:05.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.260+0000 7f116d67b640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f114c074600 0x7f114c076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.260+0000 7f116d67b640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f114c074600 0x7f114c076b10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1158034000 tx=0x7f115802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:05.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.264+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f116405e620 con 0x7f11680b4ca0 2024-04-25T15:23:05.405 INFO:teuthology.orchestra.run.smithi177.stderr:cp: cannot open '/usr/sbin/autrace' for reading: Permission denied 2024-04-25T15:23:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.486+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f11680b1cc0 con 0x7f11680b4ca0 2024-04-25T15:23:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.486+0000 7f115e7fc640 1 -- 172.21.15.146:0/3051506138 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7f1164062920 con 0x7f11680b4ca0 2024-04-25T15:23:05.487 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f114c074600 msgr2=0x7f114c076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f114c074600 0x7f114c076b10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1158034000 tx=0x7f115802e000 comp rx=0 tx=0).stop 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 msgr2=0x7f11681d8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 0x7f11681d8760 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f116400c3c0 tx=0x7f116400c880 comp rx=0 tx=0).stop 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 shutdown_connections 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f114c074600 0x7f114c076b10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11681d8ca0 0x7f11680b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11680ce6f0 0x7f11681f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 --2- 172.21.15.146:0/3051506138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11680b4ca0 0x7f11681d8760 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 >> 172.21.15.146:0/3051506138 conn(0x7f11680b5430 msgr2=0x7f11680b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:05.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 shutdown_connections 2024-04-25T15:23:05.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.489+0000 7f11700e0640 1 -- 172.21.15.146:0/3051506138 wait complete. 2024-04-25T15:23:05.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:23:05.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.623+0000 7fd3ba4aa640 1 Processor -- start 2024-04-25T15:23:05.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.623+0000 7fd3ba4aa640 1 -- start start 2024-04-25T15:23:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3ba4aa640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b4164a30 0x7fd3b4164e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3ba4aa640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3b406c8c0 0x7fd3b406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3ba4aa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b40d4930 con 0x7fd3b40bb690 2024-04-25T15:23:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3ba4aa640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd3b4165a50 con 0x7fd3b406c8c0 2024-04-25T15:23:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3ba4aa640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b40d4c10 con 0x7fd3b4164a30 2024-04-25T15:23:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3b8a47640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3b406c8c0 0x7fd3b406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3b8a47640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3b406c8c0 0x7fd3b406ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37344/0 (socket says 172.21.15.146:37344) 2024-04-25T15:23:05.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3b3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd3b40bb690 0x7fd3b40d3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50858/0 (socket says 172.21.15.146:50858) 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3b8a47640 1 -- 172.21.15.146:0/33878214 learned_addr learned my addr 172.21.15.146:0/33878214 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3b37fe640 1 --2- 172.21.15.146:0/33878214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b4164a30 0x7fd3b4164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.625+0000 7fd3b8a47640 1 -- 172.21.15.146:0/33878214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b4164a30 msgr2=0x7fd3b4164e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.626+0000 7fd3b8a47640 1 --2- 172.21.15.146:0/33878214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b4164a30 0x7fd3b4164e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.626+0000 7fd3b8a47640 1 -- 172.21.15.146:0/33878214 >> v1:172.21.15.146:6789/0 conn(0x7fd3b40bb690 legacy=0x7fd3b40d3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.626+0000 7fd3b8a47640 1 -- 172.21.15.146:0/33878214 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b40d4ef0 con 0x7fd3b406c8c0 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.626+0000 7fd3b2ffd640 1 -- 172.21.15.146:0/33878214 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037229751 0 0) 0x7fd3b40d4930 con 0x7fd3b40bb690 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.626+0000 7fd3b8a47640 1 --2- 172.21.15.146:0/33878214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b406c8c0 0x7fd3b406ccd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009e50 tx=0x7fd3ac02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f952b79765f2643 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.627+0000 7fd3b2ffd640 1 -- 172.21.15.146:0/33878214 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac036070 con 0x7fd3b406c8c0 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.627+0000 7fd3b2ffd640 1 -- 172.21.15.146:0/33878214 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac031600 con 0x7fd3b406c8c0 2024-04-25T15:23:05.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.627+0000 7fd3b2ffd640 1 -- 172.21.15.146:0/33878214 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0319d0 con 0x7fd3b406c8c0 2024-04-25T15:23:05.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/33878214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b406c8c0 msgr2=0x7fd3b406ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/33878214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b406c8c0 0x7fd3b406ccd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009e50 tx=0x7fd3ac02a680 comp rx=0 tx=0).stop 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/33878214 shutdown_connections 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/33878214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b406c8c0 0x7fd3b406ccd0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/33878214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b4164a30 0x7fd3b4164e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/33878214 >> 172.21.15.146:0/33878214 conn(0x7fd3b40d2450 msgr2=0x7fd3b40d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/33878214 shutdown_connections 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.628+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/33878214 wait complete. 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.630+0000 7fd3ba4aa640 1 Processor -- start 2024-04-25T15:23:05.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.631+0000 7fd3ba4aa640 1 -- start start 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3ba4aa640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b406c8c0 0x7fd3b41fb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3ba4aa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b40bb690 0x7fd3b415b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3ba4aa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 0x7fd3b41685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3ba4aa640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b40d4650 con 0x7fd3b4164a30 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3ba4aa640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b40d44d0 con 0x7fd3b40bb690 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3ba4aa640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b4165380 con 0x7fd3b406c8c0 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3b8a47640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 0x7fd3b41685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3b8a47640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 0x7fd3b41685d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37348/0 (socket says 172.21.15.146:37348) 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3b8a47640 1 -- 172.21.15.146:0/534313401 learned_addr learned my addr 172.21.15.146:0/534313401 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3b3fff640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b406c8c0 0x7fd3b41fb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.632+0000 7fd3b37fe640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b40bb690 0x7fd3b415b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b8a47640 1 -- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b406c8c0 msgr2=0x7fd3b41fb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b8a47640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b406c8c0 0x7fd3b41fb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b8a47640 1 -- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b40bb690 msgr2=0x7fd3b415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b8a47640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b40bb690 0x7fd3b415b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b8a47640 1 -- 172.21.15.146:0/534313401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b41fe310 con 0x7fd3b4164a30 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b37fe640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b40bb690 0x7fd3b415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b8a47640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 0x7fd3b41685d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac034040 tx=0x7fd3ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac041020 con 0x7fd3b4164a30 2024-04-25T15:23:05.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac037040 con 0x7fd3b4164a30 2024-04-25T15:23:05.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac036050 con 0x7fd3b4164a30 2024-04-25T15:23:05.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b41fe5f0 con 0x7fd3b4164a30 2024-04-25T15:23:05.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.633+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b41febb0 con 0x7fd3b4164a30 2024-04-25T15:23:05.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.635+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b406a0e0 con 0x7fd3b4164a30 2024-04-25T15:23:05.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.636+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fd3ac007c50 con 0x7fd3b4164a30 2024-04-25T15:23:05.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.637+0000 7fd3b0ff9640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3a0074600 0x7fd3a0076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:05.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.637+0000 7fd3b3fff640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3a0074600 0x7fd3a0076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:05.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.637+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fd3ac0b5c70 con 0x7fd3b4164a30 2024-04-25T15:23:05.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.637+0000 7fd3b3fff640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3a0074600 0x7fd3a0076b10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4007fd0 tx=0x7fd3a4007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:05.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.643+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd3ac04b060 con 0x7fd3b4164a30 2024-04-25T15:23:05.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.867+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd3b40742f0 con 0x7fd3b4164a30 2024-04-25T15:23:05.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.867+0000 7fd3b0ff9640 1 -- 172.21.15.146:0/534313401 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v57) ==== 80+0+122 (secure 0 0 0) 0x7fd3ac07fa00 con 0x7fd3b4164a30 2024-04-25T15:23:05.868 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6836/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:23:05.868 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6834/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:23:05.869 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.869+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3a0074600 msgr2=0x7fd3a0076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3a0074600 0x7fd3a0076b10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4007fd0 tx=0x7fd3a4007de0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 msgr2=0x7fd3b41685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 0x7fd3b41685d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac034040 tx=0x7fd3ac004a40 comp rx=0 tx=0).stop 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 shutdown_connections 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3a0074600 0x7fd3a0076b10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3b4164a30 0x7fd3b41685d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b40bb690 0x7fd3b415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 --2- 172.21.15.146:0/534313401 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b406c8c0 0x7fd3b41fb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 >> 172.21.15.146:0/534313401 conn(0x7fd3b40d2450 msgr2=0x7fd3b40d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 shutdown_connections 2024-04-25T15:23:05.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:05.870+0000 7fd3ba4aa640 1 -- 172.21.15.146:0/534313401 wait complete. 2024-04-25T15:23:05.882 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-04-25T15:23:05.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:05.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-25T15:23:05.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi146.stderr:ceph-fuse[59786]: fuse finished with error 0 and tester_r 0 2024-04-25T15:23:05.966 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:23:05.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:05.967 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T15:23:06.141 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:23:06.141 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--timeout=1', '--', 'bash', '-c', Raw("'\n set -ex\n pushd `mktemp -d -p .`\n cp -a /usr .\n popd\n '")], check_status=True, hostname='smithi177') 2024-04-25T15:23:06.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:23:06 stdin-killer INFO: expiration expected; waiting 1 seconds for command to complete 2024-04-25T15:23:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 137 2024-04-25T15:23:07.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-25T15:23:07 stdin-killer ERROR: timeout expired: sending signal 9 to command and myself 2024-04-25T15:23:07.143 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 6: 44816 Killed ( cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=1 -- bash -c ' 2024-04-25T15:23:07.143 INFO:teuthology.orchestra.run.smithi177.stderr: set -ex 2024-04-25T15:23:07.143 INFO:teuthology.orchestra.run.smithi177.stderr: pushd `mktemp -d -p .` 2024-04-25T15:23:07.143 INFO:teuthology.orchestra.run.smithi177.stderr: cp -a /usr . 2024-04-25T15:23:07.144 INFO:teuthology.orchestra.run.smithi177.stderr: popd 2024-04-25T15:23:07.144 INFO:teuthology.orchestra.run.smithi177.stderr: ' ) 2024-04-25T15:23:07.144 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:23:07.144 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:07.198 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:23:07.198 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:07.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.366+0000 7f3af9ef0640 1 Processor -- start 2024-04-25T15:23:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.366+0000 7f3af9ef0640 1 -- start start 2024-04-25T15:23:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af9ef0640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3af4153ca0 0x7f3af415dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af9ef0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 0x7f3af40d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af9ef0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af40d1410 con 0x7f3af40cbc90 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af9ef0640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f3af40cfd10 con 0x7f3af4153ca0 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af9ef0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3af40cf420 con 0x7f3af40d0ac0 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af2ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3af4153ca0 0x7f3af415dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 0x7f3af40d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 0x7f3af40d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:52680/0 (socket says 172.21.15.146:52680) 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af3fff640 1 -- 172.21.15.146:0/522382415 learned_addr learned my addr 172.21.15.146:0/522382415 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.367+0000 7f3af37fe640 1 --1- 172.21.15.146:0/522382415 >> v1:172.21.15.177:6789/0 conn(0x7f3af40cbc90 0x7f3af40d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:36776/0 (socket says 172.21.15.146:36776) 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af3fff640 1 -- 172.21.15.146:0/522382415 >> v2:172.21.15.177:3300/0 conn(0x7f3af4153ca0 msgr2=0x7f3af415dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af3fff640 1 --2- 172.21.15.146:0/522382415 >> v2:172.21.15.177:3300/0 conn(0x7f3af4153ca0 0x7f3af415dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af3fff640 1 -- 172.21.15.146:0/522382415 >> v1:172.21.15.177:6789/0 conn(0x7f3af40cbc90 legacy=0x7f3af40d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af3fff640 1 -- 172.21.15.146:0/522382415 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af40cc270 con 0x7f3af40d0ac0 2024-04-25T15:23:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af27fc640 1 -- 172.21.15.146:0/522382415 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433910348 0 0) 0x7f3ae4001110 con 0x7f3af40cbc90 2024-04-25T15:23:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af2ffd640 1 --2- 172.21.15.146:0/522382415 >> v2:172.21.15.177:3300/0 conn(0x7f3af4153ca0 0x7f3af415dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.368+0000 7f3af3fff640 1 --2- 172.21.15.146:0/522382415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 0x7f3af40d0ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009e50 tx=0x7f3aec02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ee257886039e0f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af27fc640 1 -- 172.21.15.146:0/522382415 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec036070 con 0x7f3af40d0ac0 2024-04-25T15:23:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af27fc640 1 -- 172.21.15.146:0/522382415 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aec031600 con 0x7f3af40d0ac0 2024-04-25T15:23:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af27fc640 1 -- 172.21.15.146:0/522382415 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0319d0 con 0x7f3af40d0ac0 2024-04-25T15:23:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 -- 172.21.15.146:0/522382415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 msgr2=0x7f3af40d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/522382415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 0x7f3af40d0ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009e50 tx=0x7f3aec02a710 comp rx=0 tx=0).stop 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 -- 172.21.15.146:0/522382415 shutdown_connections 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/522382415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40d0ac0 0x7f3af40d0ed0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/522382415 >> v2:172.21.15.177:3300/0 conn(0x7f3af4153ca0 0x7f3af415dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 -- 172.21.15.146:0/522382415 >> 172.21.15.146:0/522382415 conn(0x7f3af40ce0c0 msgr2=0x7f3af40ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 -- 172.21.15.146:0/522382415 shutdown_connections 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.369+0000 7f3af9ef0640 1 -- 172.21.15.146:0/522382415 wait complete. 2024-04-25T15:23:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.371+0000 7f3af9ef0640 1 Processor -- start 2024-04-25T15:23:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.371+0000 7f3af9ef0640 1 -- start start 2024-04-25T15:23:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.372+0000 7f3af9ef0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40cbc90 0x7f3af416a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.372+0000 7f3af9ef0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 0x7f3af414a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.372+0000 7f3af9ef0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3af4153ca0 0x7f3af414a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.372+0000 7f3af9ef0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3af415e1a0 con 0x7f3af40d0ac0 2024-04-25T15:23:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.372+0000 7f3af9ef0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3af40cf9a0 con 0x7f3af4153ca0 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.372+0000 7f3af9ef0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f3af41540b0 con 0x7f3af40cbc90 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 0x7f3af414a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 0x7f3af414a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37354/0 (socket says 172.21.15.146:37354) 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 -- 172.21.15.146:0/3042168390 learned_addr learned my addr 172.21.15.146:0/3042168390 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af37fe640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40cbc90 0x7f3af416a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af3fff640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3af4153ca0 0x7f3af414a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 -- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40cbc90 msgr2=0x7f3af416a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40cbc90 0x7f3af416a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 -- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3af4153ca0 msgr2=0x7f3af414a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3af4153ca0 0x7f3af414a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 -- 172.21.15.146:0/3042168390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af4145b00 con 0x7f3af40d0ac0 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af37fe640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40cbc90 0x7f3af416a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af2ffd640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 0x7f3af414a0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800e8d0 tx=0x7f3ae800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8017070 con 0x7f3af40d0ac0 2024-04-25T15:23:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af4145de0 con 0x7f3af40d0ac0 2024-04-25T15:23:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae800c940 con 0x7f3af40d0ac0 2024-04-25T15:23:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae800cd30 con 0x7f3af40d0ac0 2024-04-25T15:23:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.373+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af41463a0 con 0x7f3af40d0ac0 2024-04-25T15:23:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.375+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af406a0e0 con 0x7f3af40d0ac0 2024-04-25T15:23:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.375+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f3ae801a070 con 0x7f3af40d0ac0 2024-04-25T15:23:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.375+0000 7f3adbfff640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ae00745d0 0x7f3ae0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.375+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f3ae8095030 con 0x7f3af40d0ac0 2024-04-25T15:23:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.375+0000 7f3af37fe640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ae00745d0 0x7f3ae0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.376+0000 7f3af37fe640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ae00745d0 0x7f3ae0076ae0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000f60 tx=0x7f3ae4007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.380+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3ae805ee50 con 0x7f3af40d0ac0 2024-04-25T15:23:07.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.570+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3af40711c0 con 0x7f3af40d0ac0 2024-04-25T15:23:07.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.571+0000 7f3adbfff640 1 -- 172.21.15.146:0/3042168390 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7f3ae8063150 con 0x7f3af40d0ac0 2024-04-25T15:23:07.572 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:07.572 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ae00745d0 msgr2=0x7f3ae0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ae00745d0 0x7f3ae0076ae0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000f60 tx=0x7f3ae4007ca0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 msgr2=0x7f3af414a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 0x7f3af414a0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800e8d0 tx=0x7f3ae800ed90 comp rx=0 tx=0).stop 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 shutdown_connections 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3ae00745d0 0x7f3ae0076ae0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3af4153ca0 0x7f3af414a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3af40d0ac0 0x7f3af414a0a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 --2- 172.21.15.146:0/3042168390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3af40cbc90 0x7f3af416a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 >> 172.21.15.146:0/3042168390 conn(0x7f3af40ce0c0 msgr2=0x7f3af415f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:07.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 shutdown_connections 2024-04-25T15:23:07.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.573+0000 7f3af9ef0640 1 -- 172.21.15.146:0/3042168390 wait complete. 2024-04-25T15:23:07.585 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:23:07.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:23:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.699+0000 7f4e0c813640 1 Processor -- start 2024-04-25T15:23:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.699+0000 7f4e0c813640 1 -- start start 2024-04-25T15:23:07.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0c813640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e0806f2a0 0x7f4e080cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0c813640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0806f7b0 con 0x7f4e08160f10 2024-04-25T15:23:07.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0c813640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0809b700 con 0x7f4e08160b00 2024-04-25T15:23:07.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0c813640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4e080b1f30 con 0x7f4e0806f2a0 2024-04-25T15:23:07.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e06d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4e08160f10 0x7f4e0815cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50866/0 (socket says 172.21.15.146:50866) 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e06d9d640 1 -- 172.21.15.146:0/2909263876 learned_addr learned my addr 172.21.15.146:0/2909263876 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0659c640 1 --2- 172.21.15.146:0/2909263876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e0806f2a0 0x7f4e080cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724592167 0 0) 0x7f4e0806f7b0 con 0x7f4e08160f10 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4df0003520 con 0x7f4e08160f10 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050888134 0 0) 0x7f4e0809b700 con 0x7f4e08160b00 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e0806f7b0 con 0x7f4e08160b00 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0659c640 1 -- 172.21.15.146:0/2909263876 >> v1:172.21.15.177:6789/0 conn(0x7f4e08160b00 legacy=0x7f4e0815c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0659c640 1 -- 172.21.15.146:0/2909263876 >> v1:172.21.15.146:6789/0 conn(0x7f4e08160f10 legacy=0x7f4e0815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.701+0000 7f4e0659c640 1 -- 172.21.15.146:0/2909263876 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0809b9e0 con 0x7f4e0806f2a0 2024-04-25T15:23:07.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.702+0000 7f4e0659c640 1 --2- 172.21.15.146:0/2909263876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e0806f2a0 0x7f4e080cfa30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097d0 tx=0x7f4df802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa67000af6ce6029 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:07.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.702+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8037070 con 0x7f4e0806f2a0 2024-04-25T15:23:07.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.702+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df8032440 con 0x7f4e0806f2a0 2024-04-25T15:23:07.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.702+0000 7f4e0559a640 1 -- 172.21.15.146:0/2909263876 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80327f0 con 0x7f4e0806f2a0 2024-04-25T15:23:07.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 -- 172.21.15.146:0/2909263876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e0806f2a0 msgr2=0x7f4e080cfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 --2- 172.21.15.146:0/2909263876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e0806f2a0 0x7f4e080cfa30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097d0 tx=0x7f4df802a210 comp rx=0 tx=0).stop 2024-04-25T15:23:07.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 -- 172.21.15.146:0/2909263876 shutdown_connections 2024-04-25T15:23:07.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 --2- 172.21.15.146:0/2909263876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e0806f2a0 0x7f4e080cfa30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 -- 172.21.15.146:0/2909263876 >> 172.21.15.146:0/2909263876 conn(0x7f4e080b7490 msgr2=0x7f4e080b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:07.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 -- 172.21.15.146:0/2909263876 shutdown_connections 2024-04-25T15:23:07.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.703+0000 7f4e0c813640 1 -- 172.21.15.146:0/2909263876 wait complete. 2024-04-25T15:23:07.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.705+0000 7f4e0c813640 1 Processor -- start 2024-04-25T15:23:07.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.705+0000 7f4e0c813640 1 -- start start 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.706+0000 7f4e0c813640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 0x7f4e0809d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.706+0000 7f4e0c813640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e08160b00 0x7f4e081581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.706+0000 7f4e0c813640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e08160f10 0x7f4e0815b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.706+0000 7f4e0c813640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4e080cff70 con 0x7f4e0806f2a0 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.706+0000 7f4e0c813640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0815d260 con 0x7f4e08160b00 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.706+0000 7f4e0c813640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4e080d28b0 con 0x7f4e08160f10 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 0x7f4e0809d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 0x7f4e0809d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37358/0 (socket says 172.21.15.146:37358) 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 -- 172.21.15.146:0/3955062000 learned_addr learned my addr 172.21.15.146:0/3955062000 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:07.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e05d9b640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e08160b00 0x7f4e081581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e06d9d640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e08160f10 0x7f4e0815b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 -- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e08160f10 msgr2=0x7f4e0815b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e08160f10 0x7f4e0815b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 -- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e08160b00 msgr2=0x7f4e081581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e08160b00 0x7f4e081581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 -- 172.21.15.146:0/3955062000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e080d04d0 con 0x7f4e0806f2a0 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0659c640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 0x7f4e0809d070 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f4df8035040 tx=0x7f4df80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8041020 con 0x7f4e0806f2a0 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df803d6e0 con 0x7f4e0806f2a0 2024-04-25T15:23:07.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8037040 con 0x7f4e0806f2a0 2024-04-25T15:23:07.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e080d07b0 con 0x7f4e0806f2a0 2024-04-25T15:23:07.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.707+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e080d0d70 con 0x7f4e0806f2a0 2024-04-25T15:23:07.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.709+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0806a0e0 con 0x7f4e0806f2a0 2024-04-25T15:23:07.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.710+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4df8044020 con 0x7f4e0806f2a0 2024-04-25T15:23:07.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.710+0000 7f4df6ffd640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4de80745d0 0x7f4de8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:07.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.710+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f4df80b5c60 con 0x7f4e0806f2a0 2024-04-25T15:23:07.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.710+0000 7f4e05d9b640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4de80745d0 0x7f4de8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:07.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.711+0000 7f4e05d9b640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4de80745d0 0x7f4de8076ae0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc007fd0 tx=0x7f4dfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:07.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.715+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4df804b030 con 0x7f4e0806f2a0 2024-04-25T15:23:07.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.900+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e080c5f30 con 0x7f4e0806f2a0 2024-04-25T15:23:07.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.901+0000 7f4df6ffd640 1 -- 172.21.15.146:0/3955062000 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7f4df807fa80 con 0x7f4e0806f2a0 2024-04-25T15:23:07.902 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4de80745d0 msgr2=0x7f4de8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4de80745d0 0x7f4de8076ae0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc007fd0 tx=0x7f4dfc009040 comp rx=0 tx=0).stop 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 msgr2=0x7f4e0809d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 0x7f4e0809d070 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f4df8035040 tx=0x7f4df80046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 shutdown_connections 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4de80745d0 0x7f4de8076ae0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e08160f10 0x7f4e0815b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e08160b00 0x7f4e081581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 --2- 172.21.15.146:0/3955062000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e0806f2a0 0x7f4e0809d070 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 >> 172.21.15.146:0/3955062000 conn(0x7f4e080b7490 msgr2=0x7f4e0815a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 shutdown_connections 2024-04-25T15:23:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:07.904+0000 7f4e0c813640 1 -- 172.21.15.146:0/3955062000 wait complete. 2024-04-25T15:23:07.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:23:08.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.031+0000 7ff81d670640 1 Processor -- start 2024-04-25T15:23:08.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.031+0000 7ff81d670640 1 -- start start 2024-04-25T15:23:08.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff81d670640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff818160b00 0x7ff818160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff81d670640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff81815c440 0x7ff81815c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff81d670640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81806f7b0 con 0x7ff81806f2a0 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff81d670640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff8180b1f30 con 0x7ff81815c440 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff81d670640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7ff81809b700 con 0x7ff818160b00 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff8177fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff81815c440 0x7ff81815c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff8177fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff81815c440 0x7ff81815c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37372/0 (socket says 172.21.15.146:37372) 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff8177fe640 1 -- 172.21.15.146:0/1589495223 learned_addr learned my addr 172.21.15.146:0/1589495223 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.032+0000 7ff8167fc640 1 --2- 172.21.15.146:0/1589495223 >> v2:172.21.15.177:3300/0 conn(0x7ff818160b00 0x7ff818160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.033+0000 7ff8177fe640 1 -- 172.21.15.146:0/1589495223 >> v2:172.21.15.177:3300/0 conn(0x7ff818160b00 msgr2=0x7ff818160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.033+0000 7ff8177fe640 1 --2- 172.21.15.146:0/1589495223 >> v2:172.21.15.177:3300/0 conn(0x7ff818160b00 0x7ff818160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.033+0000 7ff8177fe640 1 -- 172.21.15.146:0/1589495223 >> v1:172.21.15.177:6789/0 conn(0x7ff81806f2a0 legacy=0x7ff8180cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.033+0000 7ff8177fe640 1 -- 172.21.15.146:0/1589495223 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81809b9e0 con 0x7ff81815c440 2024-04-25T15:23:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.033+0000 7ff815ffb640 1 -- 172.21.15.146:0/1589495223 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656492723 0 0) 0x7ff81806f7b0 con 0x7ff81806f2a0 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.034+0000 7ff8177fe640 1 --2- 172.21.15.146:0/1589495223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff81815c850 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff81806feb0 tx=0x7ff81002c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f920c15fc83eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.034+0000 7ff815ffb640 1 -- 172.21.15.146:0/1589495223 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81000b2e0 con 0x7ff81815c440 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.034+0000 7ff815ffb640 1 -- 172.21.15.146:0/1589495223 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff810037070 con 0x7ff81815c440 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.034+0000 7ff815ffb640 1 -- 172.21.15.146:0/1589495223 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81000b480 con 0x7ff81815c440 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 -- 172.21.15.146:0/1589495223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 msgr2=0x7ff81815c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 --2- 172.21.15.146:0/1589495223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff81815c850 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff81806feb0 tx=0x7ff81002c940 comp rx=0 tx=0).stop 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 -- 172.21.15.146:0/1589495223 shutdown_connections 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 --2- 172.21.15.146:0/1589495223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff81815c850 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 --2- 172.21.15.146:0/1589495223 >> v2:172.21.15.177:3300/0 conn(0x7ff818160b00 0x7ff818160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 -- 172.21.15.146:0/1589495223 >> 172.21.15.146:0/1589495223 conn(0x7ff8180b7490 msgr2=0x7ff8180b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:08.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 -- 172.21.15.146:0/1589495223 shutdown_connections 2024-04-25T15:23:08.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.035+0000 7ff81d670640 1 -- 172.21.15.146:0/1589495223 wait complete. 2024-04-25T15:23:08.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.037+0000 7ff81d670640 1 Processor -- start 2024-04-25T15:23:08.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.037+0000 7ff81d670640 1 -- start start 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff81d670640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff81806f2a0 0x7ff818146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff81d670640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff8180c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff81d670640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff818160b00 0x7ff818149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff81d670640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff81815cf10 con 0x7ff81815c440 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff81d670640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff81815cd90 con 0x7ff818160b00 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff81d670640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ff81815d090 con 0x7ff81806f2a0 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff8180c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff8180c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37376/0 (socket says 172.21.15.146:37376) 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 -- 172.21.15.146:0/1649914604 learned_addr learned my addr 172.21.15.146:0/1649914604 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:08.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8177fe640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff818160b00 0x7ff818149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff816ffd640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff81806f2a0 0x7ff818146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 -- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff81806f2a0 msgr2=0x7ff818146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff81806f2a0 0x7ff818146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 -- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff818160b00 msgr2=0x7ff818149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff818160b00 0x7ff818149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 -- 172.21.15.146:0/1649914604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8180d2210 con 0x7ff81815c440 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.038+0000 7ff8167fc640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff8180c5640 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00e8a0 tx=0x7ff80c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.039+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c017070 con 0x7ff81815c440 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.039+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80c00c940 con 0x7ff81815c440 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.039+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8180b0870 con 0x7ff81815c440 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.039+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c00cd30 con 0x7ff81815c440 2024-04-25T15:23:08.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.039+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff818149a90 con 0x7ff81815c440 2024-04-25T15:23:08.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.040+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81806a0e0 con 0x7ff81815c440 2024-04-25T15:23:08.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.040+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7ff80c01a070 con 0x7ff81815c440 2024-04-25T15:23:08.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.041+0000 7ff7ff7fe640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff8040745d0 0x7ff804076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.041+0000 7ff816ffd640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff8040745d0 0x7ff804076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.041+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7ff80c0957d0 con 0x7ff81815c440 2024-04-25T15:23:08.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.041+0000 7ff816ffd640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff8040745d0 0x7ff804076ae0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff808007fd0 tx=0x7ff808009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.046+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff80c05fcc0 con 0x7ff81815c440 2024-04-25T15:23:08.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.233+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff8180c5f30 con 0x7ff81815c440 2024-04-25T15:23:08.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.233+0000 7ff7ff7fe640 1 -- 172.21.15.146:0/1649914604 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7ff80c05f5c0 con 0x7ff81815c440 2024-04-25T15:23:08.234 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:23:08.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff8040745d0 msgr2=0x7ff804076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff8040745d0 0x7ff804076ae0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff808007fd0 tx=0x7ff808009040 comp rx=0 tx=0).stop 2024-04-25T15:23:08.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 msgr2=0x7ff8180c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff8180c5640 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00e8a0 tx=0x7ff80c00ed60 comp rx=0 tx=0).stop 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 shutdown_connections 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ff8040745d0 0x7ff804076ae0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff818160b00 0x7ff818149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff81815c440 0x7ff8180c5640 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 --2- 172.21.15.146:0/1649914604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff81806f2a0 0x7ff818146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 >> 172.21.15.146:0/1649914604 conn(0x7ff8180b7490 msgr2=0x7ff818070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 shutdown_connections 2024-04-25T15:23:08.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.235+0000 7ff81d670640 1 -- 172.21.15.146:0/1649914604 wait complete. 2024-04-25T15:23:08.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:23:08.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.364+0000 7fa03c3a8640 1 Processor -- start 2024-04-25T15:23:08.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.364+0000 7fa03c3a8640 1 -- start start 2024-04-25T15:23:08.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.365+0000 7fa03c3a8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0340cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.365+0000 7fa03c3a8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa034160b00 0x7fa034160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.365+0000 7fa03c3a8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa03415c440 0x7fa03415c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.365+0000 7fa03c3a8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa0340b1f30 con 0x7fa034160b00 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.365+0000 7fa03c3a8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fa03406f7b0 con 0x7fa03415c440 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.365+0000 7fa03c3a8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa03409b700 con 0x7fa03406f2a0 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa034160b00 0x7fa034160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa034160b00 0x7fa034160f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37392/0 (socket says 172.21.15.146:37392) 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 -- 172.21.15.146:0/2597659870 learned_addr learned my addr 172.21.15.146:0/2597659870 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa03a144640 1 --2- 172.21.15.146:0/2597659870 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0340cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa03a945640 1 --2- 172.21.15.146:0/2597659870 >> v2:172.21.15.177:3300/0 conn(0x7fa03415c440 0x7fa03415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 -- 172.21.15.146:0/2597659870 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 msgr2=0x7fa0340cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 --2- 172.21.15.146:0/2597659870 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0340cfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 -- 172.21.15.146:0/2597659870 >> v2:172.21.15.177:3300/0 conn(0x7fa03415c440 msgr2=0x7fa03415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 --2- 172.21.15.146:0/2597659870 >> v2:172.21.15.177:3300/0 conn(0x7fa03415c440 0x7fa03415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.366+0000 7fa039943640 1 -- 172.21.15.146:0/2597659870 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03409b9e0 con 0x7fa034160b00 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.367+0000 7fa039943640 1 --2- 172.21.15.146:0/2597659870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa034160f10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa03409b340 tx=0x7fa02402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20864b341f7ba32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.367+0000 7fa039142640 1 -- 172.21.15.146:0/2597659870 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024037070 con 0x7fa034160b00 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.367+0000 7fa039142640 1 -- 172.21.15.146:0/2597659870 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa024032440 con 0x7fa034160b00 2024-04-25T15:23:08.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.367+0000 7fa039142640 1 -- 172.21.15.146:0/2597659870 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024032810 con 0x7fa034160b00 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 -- 172.21.15.146:0/2597659870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 msgr2=0x7fa034160f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/2597659870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa034160f10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa03409b340 tx=0x7fa02402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 -- 172.21.15.146:0/2597659870 shutdown_connections 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/2597659870 >> v2:172.21.15.177:3300/0 conn(0x7fa03415c440 0x7fa03415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/2597659870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa034160f10 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/2597659870 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0340cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 -- 172.21.15.146:0/2597659870 >> 172.21.15.146:0/2597659870 conn(0x7fa0340b7490 msgr2=0x7fa0340b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:08.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 -- 172.21.15.146:0/2597659870 shutdown_connections 2024-04-25T15:23:08.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.368+0000 7fa03c3a8640 1 -- 172.21.15.146:0/2597659870 wait complete. 2024-04-25T15:23:08.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.370+0000 7fa03c3a8640 1 Processor -- start 2024-04-25T15:23:08.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.370+0000 7fa03c3a8640 1 -- start start 2024-04-25T15:23:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.371+0000 7fa03c3a8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0341511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.371+0000 7fa03c3a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03415c440 0x7fa03409fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.371+0000 7fa03c3a8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa0340a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.371+0000 7fa03c3a8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0340cff70 con 0x7fa034160b00 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.371+0000 7fa03c3a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa03415cd90 con 0x7fa03415c440 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.371+0000 7fa03c3a8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa03414e850 con 0x7fa03406f2a0 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa0340a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa0340a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37398/0 (socket says 172.21.15.146:37398) 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 -- 172.21.15.146:0/1470695756 learned_addr learned my addr 172.21.15.146:0/1470695756 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a144640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0341511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa039943640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03415c440 0x7fa03409fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 -- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 msgr2=0x7fa0341511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0341511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 -- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03415c440 msgr2=0x7fa03409fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03415c440 0x7fa03409fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 -- 172.21.15.146:0/1470695756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03409c7b0 con 0x7fa034160b00 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.372+0000 7fa03a945640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa0340a0140 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00efc0 tx=0x7fa02c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.373+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c018020 con 0x7fa034160b00 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.373+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa02c014480 con 0x7fa034160b00 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.373+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03409ca90 con 0x7fa034160b00 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.373+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c009e60 con 0x7fa034160b00 2024-04-25T15:23:08.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.373+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03409d050 con 0x7fa034160b00 2024-04-25T15:23:08.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.374+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa03406a0e0 con 0x7fa034160b00 2024-04-25T15:23:08.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.374+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fa02c005d10 con 0x7fa034160b00 2024-04-25T15:23:08.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.375+0000 7fa02affd640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa018074600 0x7fa018076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.375+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fa02c096220 con 0x7fa034160b00 2024-04-25T15:23:08.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.375+0000 7fa03a144640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa018074600 0x7fa018076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.375+0000 7fa03a144640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa018074600 0x7fa018076b10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa030009a20 tx=0x7fa030009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.381+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa02c022030 con 0x7fa034160b00 2024-04-25T15:23:08.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.562+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa0340c5f30 con 0x7fa034160b00 2024-04-25T15:23:08.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.562+0000 7fa02affd640 1 -- 172.21.15.146:0/1470695756 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7fa02c010040 con 0x7fa034160b00 2024-04-25T15:23:08.563 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:23:08.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa018074600 msgr2=0x7fa018076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa018074600 0x7fa018076b10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa030009a20 tx=0x7fa030009190 comp rx=0 tx=0).stop 2024-04-25T15:23:08.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 msgr2=0x7fa0340a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa0340a0140 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00efc0 tx=0x7fa02c00c490 comp rx=0 tx=0).stop 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 shutdown_connections 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa018074600 0x7fa018076b10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa034160b00 0x7fa0340a0140 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa03415c440 0x7fa03409fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 --2- 172.21.15.146:0/1470695756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa03406f2a0 0x7fa0341511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 >> 172.21.15.146:0/1470695756 conn(0x7fa0340b7490 msgr2=0x7fa034071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 shutdown_connections 2024-04-25T15:23:08.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.564+0000 7fa03c3a8640 1 -- 172.21.15.146:0/1470695756 wait complete. 2024-04-25T15:23:08.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:23:08.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.685+0000 7f0459172640 1 Processor -- start 2024-04-25T15:23:08.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.685+0000 7f0459172640 1 -- start start 2024-04-25T15:23:08.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.686+0000 7f0459172640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 0x7f04540b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.686+0000 7f0459172640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0454163eb0 0x7f04541642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.686+0000 7f0459172640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04540b8eb0 con 0x7f0454164800 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.686+0000 7f0459172640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f045408d730 con 0x7f045406ee60 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.686+0000 7f0459172640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f045406f440 con 0x7f0454163eb0 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f0452d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 0x7f04540b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f044bfff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0454163eb0 0x7f04541642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f0452d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 0x7f04540b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39532/0 (socket says 172.21.15.146:39532) 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f045359e640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f0454164800 0x7f0454160830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:36806/0 (socket says 172.21.15.146:36806) 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f0452d9d640 1 -- 172.21.15.146:0/2147830264 learned_addr learned my addr 172.21.15.146:0/2147830264 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f044bfff640 1 -- 172.21.15.146:0/2147830264 >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 msgr2=0x7f04540b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f044bfff640 1 --2- 172.21.15.146:0/2147830264 >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 0x7f04540b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f044bfff640 1 -- 172.21.15.146:0/2147830264 >> v1:172.21.15.177:6789/0 conn(0x7f0454164800 legacy=0x7f0454160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f045259c640 1 -- 172.21.15.146:0/2147830264 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599908991 0 0) 0x7f04540b8eb0 con 0x7f0454164800 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f044bfff640 1 -- 172.21.15.146:0/2147830264 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045408ce00 con 0x7f0454163eb0 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.687+0000 7f0452d9d640 1 --2- 172.21.15.146:0/2147830264 >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 0x7f04540b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.688+0000 7f044bfff640 1 --2- 172.21.15.146:0/2147830264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0454163eb0 0x7f04541642c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f045408d8b0 tx=0x7f043c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a033fc1ee92797a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.688+0000 7f045259c640 1 -- 172.21.15.146:0/2147830264 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c037070 con 0x7f0454163eb0 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.688+0000 7f045259c640 1 -- 172.21.15.146:0/2147830264 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f043c032440 con 0x7f0454163eb0 2024-04-25T15:23:08.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.688+0000 7f045259c640 1 -- 172.21.15.146:0/2147830264 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c032810 con 0x7f0454163eb0 2024-04-25T15:23:08.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 -- 172.21.15.146:0/2147830264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0454163eb0 msgr2=0x7f04541642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 --2- 172.21.15.146:0/2147830264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0454163eb0 0x7f04541642c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f045408d8b0 tx=0x7f043c02a180 comp rx=0 tx=0).stop 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 -- 172.21.15.146:0/2147830264 shutdown_connections 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 --2- 172.21.15.146:0/2147830264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0454163eb0 0x7f04541642c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 --2- 172.21.15.146:0/2147830264 >> v2:172.21.15.177:3300/0 conn(0x7f045406ee60 0x7f04540b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 -- 172.21.15.146:0/2147830264 >> 172.21.15.146:0/2147830264 conn(0x7f04540b52e0 msgr2=0x7f04540b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 -- 172.21.15.146:0/2147830264 shutdown_connections 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.689+0000 7f0459172640 1 -- 172.21.15.146:0/2147830264 wait complete. 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.691+0000 7f0459172640 1 Processor -- start 2024-04-25T15:23:08.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.691+0000 7f0459172640 1 -- start start 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0459172640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f045406ee60 0x7f0454157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0459172640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0454163eb0 0x7f04540b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0459172640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 0x7f045406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0459172640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f045408dbc0 con 0x7f0454164800 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0459172640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f045408d3c0 con 0x7f0454163eb0 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0459172640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f04540d2b70 con 0x7f045406ee60 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f045359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 0x7f045406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f045359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 0x7f045406e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37412/0 (socket says 172.21.15.146:37412) 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f045359e640 1 -- 172.21.15.146:0/2922136892 learned_addr learned my addr 172.21.15.146:0/2922136892 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:08.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f0452d9d640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f045406ee60 0x7f0454157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.692+0000 7f044bfff640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0454163eb0 0x7f04540b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f045359e640 1 -- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f045406ee60 msgr2=0x7f0454157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f045359e640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f045406ee60 0x7f0454157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f045359e640 1 -- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0454163eb0 msgr2=0x7f04540b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f045359e640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0454163eb0 0x7f04540b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f045359e640 1 -- 172.21.15.146:0/2922136892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045415cbf0 con 0x7f0454164800 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f045359e640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 0x7f045406e1b0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f044c00c700 tx=0x7f044c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c018070 con 0x7f0454164800 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044c00ce10 con 0x7f0454164800 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045415ced0 con 0x7f0454164800 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c00a880 con 0x7f0454164800 2024-04-25T15:23:08.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.693+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045415d490 con 0x7f0454164800 2024-04-25T15:23:08.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.694+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f045415c960 con 0x7f0454164800 2024-04-25T15:23:08.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.695+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f044c01b070 con 0x7f0454164800 2024-04-25T15:23:08.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.695+0000 7f044b7fe640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0434074600 0x7f0434076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:08.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.695+0000 7f0452d9d640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0434074600 0x7f0434076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:08.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.695+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f044c0949c0 con 0x7f0454164800 2024-04-25T15:23:08.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.695+0000 7f0452d9d640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0434074600 0x7f0434076b10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f04440095c0 tx=0x7f0444009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:08.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.700+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f044c096e20 con 0x7f0454164800 2024-04-25T15:23:08.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.892+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f04540d1460 con 0x7f0454164800 2024-04-25T15:23:08.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.892+0000 7f044b7fe640 1 -- 172.21.15.146:0/2922136892 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v57) ==== 80+0+122 (secure 0 0 0) 0x7f044c05e8f0 con 0x7f0454164800 2024-04-25T15:23:08.893 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6836/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:23:08.894 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6834/865508848 2024-04-26T15:22:49.831455+0000 2024-04-25T15:23:08.894 INFO:teuthology.orchestra.run.smithi146.stderr:listed 2 entries 2024-04-25T15:23:08.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0434074600 msgr2=0x7f0434076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0434074600 0x7f0434076b10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f04440095c0 tx=0x7f0444009210 comp rx=0 tx=0).stop 2024-04-25T15:23:08.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 msgr2=0x7f045406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:08.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 0x7f045406e1b0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f044c00c700 tx=0x7f044c00cbc0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 shutdown_connections 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f0434074600 0x7f0434076b10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0454164800 0x7f045406e1b0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0454163eb0 0x7f04540b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 --2- 172.21.15.146:0/2922136892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f045406ee60 0x7f0454157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 >> 172.21.15.146:0/2922136892 conn(0x7f04540b52e0 msgr2=0x7f045415e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.894+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 shutdown_connections 2024-04-25T15:23:08.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:08.895+0000 7f0459172640 1 -- 172.21.15.146:0/2922136892 wait complete. 2024-04-25T15:23:08.907 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi177.front.sepia.ceph.com... 2024-04-25T15:23:08.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:08.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-25T15:23:08.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi177.stderr:ceph-fuse[44680]: fuse finished with error 0 and tester_r 0 2024-04-25T15:23:08.977 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:23:08.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:08.978 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T15:23:09.155 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:23:09.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.265+0000 7fc907215640 1 Processor -- start 2024-04-25T15:23:09.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.265+0000 7fc907215640 1 -- start start 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc907215640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc9000d1cc0 0x7fc9000d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc907215640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc900161b30 0x7fc900161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc907215640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9000d1420 con 0x7fc9000cf1d0 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc907215640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc9000ced40 con 0x7fc900161b30 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc907215640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc9000d1700 con 0x7fc9000d1cc0 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc9057b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc900161b30 0x7fc900161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc904fb1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc9000cf1d0 0x7fc9000b7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50868/0 (socket says 172.21.15.146:50868) 2024-04-25T15:23:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc904fb1640 1 -- 172.21.15.146:0/666719908 learned_addr learned my addr 172.21.15.146:0/666719908 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc8f7fff640 1 --2- 172.21.15.146:0/666719908 >> v2:172.21.15.177:3300/0 conn(0x7fc9000d1cc0 0x7fc9000d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc9057b2640 1 -- 172.21.15.146:0/666719908 >> v2:172.21.15.177:3300/0 conn(0x7fc9000d1cc0 msgr2=0x7fc9000d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc9057b2640 1 --2- 172.21.15.146:0/666719908 >> v2:172.21.15.177:3300/0 conn(0x7fc9000d1cc0 0x7fc9000d20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc8f77fe640 1 -- 172.21.15.146:0/666719908 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352680397 0 0) 0x7fc9000d1420 con 0x7fc9000cf1d0 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc9057b2640 1 -- 172.21.15.146:0/666719908 >> v1:172.21.15.146:6789/0 conn(0x7fc9000cf1d0 legacy=0x7fc9000b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc9057b2640 1 -- 172.21.15.146:0/666719908 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9000d19e0 con 0x7fc900161b30 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.267+0000 7fc8f7fff640 1 --2- 172.21.15.146:0/666719908 >> v2:172.21.15.177:3300/0 conn(0x7fc9000d1cc0 0x7fc9000d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.268+0000 7fc9057b2640 1 --2- 172.21.15.146:0/666719908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc900161b30 0x7fc900161f40 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc9000cd680 tx=0x7fc8f802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54279e4f9977ce66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.268+0000 7fc8f77fe640 1 -- 172.21.15.146:0/666719908 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8007740 con 0x7fc900161b30 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.268+0000 7fc8f77fe640 1 -- 172.21.15.146:0/666719908 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8f80078e0 con 0x7fc900161b30 2024-04-25T15:23:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.268+0000 7fc8f77fe640 1 -- 172.21.15.146:0/666719908 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8007cb0 con 0x7fc900161b30 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 -- 172.21.15.146:0/666719908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc900161b30 msgr2=0x7fc900161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 --2- 172.21.15.146:0/666719908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc900161b30 0x7fc900161f40 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc9000cd680 tx=0x7fc8f802a680 comp rx=0 tx=0).stop 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 -- 172.21.15.146:0/666719908 shutdown_connections 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 --2- 172.21.15.146:0/666719908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc900161b30 0x7fc900161f40 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 --2- 172.21.15.146:0/666719908 >> v2:172.21.15.177:3300/0 conn(0x7fc9000d1cc0 0x7fc9000d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 -- 172.21.15.146:0/666719908 >> 172.21.15.146:0/666719908 conn(0x7fc90009b2b0 msgr2=0x7fc9000b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 -- 172.21.15.146:0/666719908 shutdown_connections 2024-04-25T15:23:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.269+0000 7fc907215640 1 -- 172.21.15.146:0/666719908 wait complete. 2024-04-25T15:23:09.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.270+0000 7fc907215640 1 Processor -- start 2024-04-25T15:23:09.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.270+0000 7fc907215640 1 -- start start 2024-04-25T15:23:09.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc907215640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 0x7fc9000c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc907215640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9000d1cc0 0x7fc90014e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc907215640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc900161b30 0x7fc900150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc907215640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9000d2610 con 0x7fc9000cf1d0 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc907215640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc9000cf5e0 con 0x7fc9000d1cc0 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc907215640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc90015e310 con 0x7fc900161b30 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 0x7fc9000c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 0x7fc9000c5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37432/0 (socket says 172.21.15.146:37432) 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 -- 172.21.15.146:0/1355850200 learned_addr learned my addr 172.21.15.146:0/1355850200 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc9057b2640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc900161b30 0x7fc900150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc8f7fff640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9000d1cc0 0x7fc90014e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 -- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc900161b30 msgr2=0x7fc900150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc900161b30 0x7fc900150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 -- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9000d1cc0 msgr2=0x7fc90014e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9000d1cc0 0x7fc90014e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.272+0000 7fc904fb1640 1 -- 172.21.15.146:0/1355850200 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90014fab0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.273+0000 7fc904fb1640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 0x7fc9000c5a00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00c440 tx=0x7fc8fc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.273+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0074e0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.273+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8fc0047e0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.273+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90014fd90 con 0x7fc9000cf1d0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.273+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc004bd0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.273+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc900150350 con 0x7fc9000cf1d0 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.274+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90006a0e0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.275+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fc8fc011070 con 0x7fc9000cf1d0 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.275+0000 7fc8f57fa640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc8ec074600 0x7fc8ec076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.275+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fc8fc095000 con 0x7fc9000cf1d0 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.275+0000 7fc8f7fff640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc8ec074600 0x7fc8ec076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.276+0000 7fc8f7fff640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc8ec074600 0x7fc8ec076b10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc9000cfc30 tx=0x7fc8f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.280+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8fc05eea0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.492+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9000b6fe0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.493+0000 7fc8f57fa640 1 -- 172.21.15.146:0/1355850200 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5191 (secure 0 0 0) 0x7fc8fc0631a0 con 0x7fc9000cf1d0 2024-04-25T15:23:09.494 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:09.494 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:53.856951+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6819,"qdb_cluster":[6819]},"id":3}]} 2024-04-25T15:23:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 36 2024-04-25T15:23:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc8ec074600 msgr2=0x7fc8ec076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc8ec074600 0x7fc8ec076b10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc9000cfc30 tx=0x7fc8f0008040 comp rx=0 tx=0).stop 2024-04-25T15:23:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 msgr2=0x7fc9000c5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 0x7fc9000c5a00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00c440 tx=0x7fc8fc00c900 comp rx=0 tx=0).stop 2024-04-25T15:23:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 shutdown_connections 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc8ec074600 0x7fc8ec076b10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc900161b30 0x7fc900150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9000d1cc0 0x7fc90014e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.495+0000 7fc907215640 1 --2- 172.21.15.146:0/1355850200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000cf1d0 0x7fc9000c5a00 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.496+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 >> 172.21.15.146:0/1355850200 conn(0x7fc90009b2b0 msgr2=0x7fc9000d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.496+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 shutdown_connections 2024-04-25T15:23:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.496+0000 7fc907215640 1 -- 172.21.15.146:0/1355850200 wait complete. 2024-04-25T15:23:09.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:09.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.615+0000 7fcd646b6640 1 Processor -- start 2024-04-25T15:23:09.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.615+0000 7fcd646b6640 1 -- start start 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd646b6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0cc280 0x7fcd5c070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd646b6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c1618f0 0x7fcd5c161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd646b6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5c162240 con 0x7fcd5c15e300 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd646b6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcd5c15bcd0 con 0x7fcd5c0cc280 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd646b6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5c0cd490 con 0x7fcd5c1618f0 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0cc280 0x7fcd5c070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0cc280 0x7fcd5c070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37440/0 (socket says 172.21.15.146:37440) 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 -- 172.21.15.146:0/2067128978 learned_addr learned my addr 172.21.15.146:0/2067128978 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd61c51640 1 --2- 172.21.15.146:0/2067128978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c1618f0 0x7fcd5c161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 -- 172.21.15.146:0/2067128978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c1618f0 msgr2=0x7fcd5c161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 --2- 172.21.15.146:0/2067128978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c1618f0 0x7fcd5c161d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 -- 172.21.15.146:0/2067128978 >> v1:172.21.15.177:6789/0 conn(0x7fcd5c15e300 legacy=0x7fcd5c15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.617+0000 7fcd62452640 1 -- 172.21.15.146:0/2067128978 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c0711d0 con 0x7fcd5c0cc280 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.618+0000 7fcd62452640 1 --2- 172.21.15.146:0/2067128978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c0cc280 0x7fcd5c070c90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009e50 tx=0x7fcd5802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ddb36f9aed339 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.618+0000 7fcd61450640 1 -- 172.21.15.146:0/2067128978 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58036070 con 0x7fcd5c0cc280 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.618+0000 7fcd61450640 1 -- 172.21.15.146:0/2067128978 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd58031600 con 0x7fcd5c0cc280 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.618+0000 7fcd61450640 1 -- 172.21.15.146:0/2067128978 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd580319d0 con 0x7fcd5c0cc280 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 -- 172.21.15.146:0/2067128978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c0cc280 msgr2=0x7fcd5c070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 --2- 172.21.15.146:0/2067128978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c0cc280 0x7fcd5c070c90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009e50 tx=0x7fcd5802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 -- 172.21.15.146:0/2067128978 shutdown_connections 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 --2- 172.21.15.146:0/2067128978 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c1618f0 0x7fcd5c161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 --2- 172.21.15.146:0/2067128978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c0cc280 0x7fcd5c070c90 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 -- 172.21.15.146:0/2067128978 >> 172.21.15.146:0/2067128978 conn(0x7fcd5c0ce0c0 msgr2=0x7fcd5c0ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 -- 172.21.15.146:0/2067128978 shutdown_connections 2024-04-25T15:23:09.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.619+0000 7fcd646b6640 1 -- 172.21.15.146:0/2067128978 wait complete. 2024-04-25T15:23:09.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.620+0000 7fcd646b6640 1 Processor -- start 2024-04-25T15:23:09.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.620+0000 7fcd646b6640 1 -- start start 2024-04-25T15:23:09.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd646b6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c0cc280 0x7fcd5c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd646b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 0x7fcd5c15a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd646b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd5c1618f0 0x7fcd5c15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd646b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c0cce40 con 0x7fcd5c15e300 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd646b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c0ccb60 con 0x7fcd5c1618f0 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd646b6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5c0cd120 con 0x7fcd5c0cc280 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 0x7fcd5c15a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 0x7fcd5c15a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37450/0 (socket says 172.21.15.146:37450) 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 -- 172.21.15.146:0/4142859126 learned_addr learned my addr 172.21.15.146:0/4142859126 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd62c53640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd5c1618f0 0x7fcd5c15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd62452640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c0cc280 0x7fcd5c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 -- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c0cc280 msgr2=0x7fcd5c14f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c0cc280 0x7fcd5c14f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 -- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd5c1618f0 msgr2=0x7fcd5c15a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd5c1618f0 0x7fcd5c15a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 -- 172.21.15.146:0/4142859126 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c0b7580 con 0x7fcd5c15e300 2024-04-25T15:23:09.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.622+0000 7fcd61c51640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 0x7fcd5c15a460 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c00e8a0 tx=0x7fcd4c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.623+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c017070 con 0x7fcd5c15e300 2024-04-25T15:23:09.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.623+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd4c00c940 con 0x7fcd5c15e300 2024-04-25T15:23:09.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.623+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5c14fb80 con 0x7fcd5c15e300 2024-04-25T15:23:09.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.623+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00cd30 con 0x7fcd5c15e300 2024-04-25T15:23:09.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.623+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5c150140 con 0x7fcd5c15e300 2024-04-25T15:23:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.624+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd5c06a0e0 con 0x7fcd5c15e300 2024-04-25T15:23:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.624+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fcd4c01a070 con 0x7fcd5c15e300 2024-04-25T15:23:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.625+0000 7fcd52ffd640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd400745d0 0x7fcd40076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.625+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fcd4c0957f0 con 0x7fcd5c15e300 2024-04-25T15:23:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.625+0000 7fcd62452640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd400745d0 0x7fcd40076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.625+0000 7fcd62452640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd400745d0 0x7fcd40076ae0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcd58033000 tx=0x7fcd58009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.629+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd4c05f580 con 0x7fcd5c15e300 2024-04-25T15:23:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.806+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd5c0b5140 con 0x7fcd5c15e300 2024-04-25T15:23:09.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.807+0000 7fcd52ffd640 1 -- 172.21.15.146:0/4142859126 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7fcd4c063880 con 0x7fcd5c15e300 2024-04-25T15:23:09.808 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:09.809 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:09.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd400745d0 msgr2=0x7fcd40076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd400745d0 0x7fcd40076ae0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcd58033000 tx=0x7fcd58009000 comp rx=0 tx=0).stop 2024-04-25T15:23:09.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 msgr2=0x7fcd5c15a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 0x7fcd5c15a460 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c00e8a0 tx=0x7fcd4c00ed60 comp rx=0 tx=0).stop 2024-04-25T15:23:09.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 shutdown_connections 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fcd400745d0 0x7fcd40076ae0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd5c1618f0 0x7fcd5c15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e300 0x7fcd5c15a460 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 --2- 172.21.15.146:0/4142859126 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcd5c0cc280 0x7fcd5c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 >> 172.21.15.146:0/4142859126 conn(0x7fcd5c0ce0c0 msgr2=0x7fcd5c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 shutdown_connections 2024-04-25T15:23:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.810+0000 7fcd646b6640 1 -- 172.21.15.146:0/4142859126 wait complete. 2024-04-25T15:23:09.822 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:19:34.001962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:22:17.219708+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:22:18.540792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:23:09.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:23:09.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.934+0000 7f441681a640 1 Processor -- start 2024-04-25T15:23:09.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.934+0000 7f441681a640 1 -- start start 2024-04-25T15:23:09.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f441681a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b4c30 0x7f44100b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f441681a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f44100b9110 0x7f44101600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f441681a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f44100d0a30 0x7f44100d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f441681a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f441006cc00 con 0x7f44100d0a30 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f441681a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f44100b9520 con 0x7f44100b9110 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f441681a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f44100d1380 con 0x7f44100b4c30 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f44100d0a30 0x7f44100d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f44100d0a30 0x7f44100d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37462/0 (socket says 172.21.15.146:37462) 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 -- 172.21.15.146:0/711523907 learned_addr learned my addr 172.21.15.146:0/711523907 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f440f7fe640 1 --2- 172.21.15.146:0/711523907 >> v2:172.21.15.177:3300/0 conn(0x7f44100b9110 0x7f44101600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f440ffff640 1 --2- 172.21.15.146:0/711523907 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b4c30 0x7f44100b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 -- 172.21.15.146:0/711523907 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b4c30 msgr2=0x7f44100b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:09.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 --2- 172.21.15.146:0/711523907 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b4c30 0x7f44100b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 -- 172.21.15.146:0/711523907 >> v2:172.21.15.177:3300/0 conn(0x7f44100b9110 msgr2=0x7f44101600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 --2- 172.21.15.146:0/711523907 >> v2:172.21.15.177:3300/0 conn(0x7f44100b9110 0x7f44101600d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.935+0000 7f4414db7640 1 -- 172.21.15.146:0/711523907 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441006c5f0 con 0x7f44100d0a30 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.936+0000 7f4414db7640 1 --2- 172.21.15.146:0/711523907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f44100d0e40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f441008dec0 tx=0x7f440802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769de260947c934f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.936+0000 7f440effd640 1 -- 172.21.15.146:0/711523907 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408036070 con 0x7f44100d0a30 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.936+0000 7f440effd640 1 -- 172.21.15.146:0/711523907 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4408031600 con 0x7f44100d0a30 2024-04-25T15:23:09.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.936+0000 7f440effd640 1 -- 172.21.15.146:0/711523907 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080319d0 con 0x7f44100d0a30 2024-04-25T15:23:09.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 -- 172.21.15.146:0/711523907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 msgr2=0x7f44100d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 --2- 172.21.15.146:0/711523907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f44100d0e40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f441008dec0 tx=0x7f440802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 -- 172.21.15.146:0/711523907 shutdown_connections 2024-04-25T15:23:09.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 --2- 172.21.15.146:0/711523907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f44100d0e40 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 --2- 172.21.15.146:0/711523907 >> v2:172.21.15.177:3300/0 conn(0x7f44100b9110 0x7f44101600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 --2- 172.21.15.146:0/711523907 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b4c30 0x7f44100b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 -- 172.21.15.146:0/711523907 >> 172.21.15.146:0/711523907 conn(0x7f44100b7970 msgr2=0x7f44100b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 -- 172.21.15.146:0/711523907 shutdown_connections 2024-04-25T15:23:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.937+0000 7f441681a640 1 -- 172.21.15.146:0/711523907 wait complete. 2024-04-25T15:23:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.939+0000 7f441681a640 1 Processor -- start 2024-04-25T15:23:09.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.939+0000 7f441681a640 1 -- start start 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f441681a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44100b4c30 0x7f4410153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f441681a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b9110 0x7f44100cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f441681a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f4410162b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f441681a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4410164a90 con 0x7f44100d0a30 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f441681a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f441006c890 con 0x7f44100b4c30 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f441681a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f44100d16b0 con 0x7f44100b9110 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f4414db7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f4410162b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f4414db7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f4410162b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37478/0 (socket says 172.21.15.146:37478) 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f4414db7640 1 -- 172.21.15.146:0/2620310695 learned_addr learned my addr 172.21.15.146:0/2620310695 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f440ffff640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44100b4c30 0x7f4410153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.940+0000 7f440f7fe640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b9110 0x7f44100cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f4414db7640 1 -- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b9110 msgr2=0x7f44100cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f4414db7640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b9110 0x7f44100cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f4414db7640 1 -- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44100b4c30 msgr2=0x7f4410153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f4414db7640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44100b4c30 0x7f4410153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f4414db7640 1 -- 172.21.15.146:0/2620310695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44100cc840 con 0x7f44100d0a30 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f4414db7640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f4410162b00 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f4408009f40 tx=0x7f4408004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408036050 con 0x7f44100d0a30 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4408042070 con 0x7f44100d0a30 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408009510 con 0x7f44100d0a30 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44100ccd70 con 0x7f44100d0a30 2024-04-25T15:23:09.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.941+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44100cd330 con 0x7f44100d0a30 2024-04-25T15:23:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.942+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44100ccae0 con 0x7f44100d0a30 2024-04-25T15:23:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.943+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4408007c70 con 0x7f44100d0a30 2024-04-25T15:23:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.943+0000 7f440cff9640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43fc074600 0x7f43fc076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.943+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f44080b6590 con 0x7f44100d0a30 2024-04-25T15:23:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.943+0000 7f440ffff640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43fc074600 0x7f43fc076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:09.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.944+0000 7f440ffff640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43fc074600 0x7f43fc076b10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f441008e090 tx=0x7f4400008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:09.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:09.948+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4408080320 con 0x7f44100d0a30 2024-04-25T15:23:10.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.136+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f441006a0e0 con 0x7f44100d0a30 2024-04-25T15:23:10.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.136+0000 7f440cff9640 1 -- 172.21.15.146:0/2620310695 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7f4408084620 con 0x7f44100d0a30 2024-04-25T15:23:10.137 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-25T15:23:10.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43fc074600 msgr2=0x7f43fc076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43fc074600 0x7f43fc076b10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f441008e090 tx=0x7f4400008040 comp rx=0 tx=0).stop 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 msgr2=0x7f4410162b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f4410162b00 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f4408009f40 tx=0x7f4408004a40 comp rx=0 tx=0).stop 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 shutdown_connections 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f43fc074600 0x7f43fc076b10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44100d0a30 0x7f4410162b00 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44100b9110 0x7f44100cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 --2- 172.21.15.146:0/2620310695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44100b4c30 0x7f4410153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 >> 172.21.15.146:0/2620310695 conn(0x7f44100b7970 msgr2=0x7f4410160fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:10.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 shutdown_connections 2024-04-25T15:23:10.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.138+0000 7f441681a640 1 -- 172.21.15.146:0/2620310695 wait complete. 2024-04-25T15:23:10.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:23:10.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.263+0000 7f34b820c640 1 Processor -- start 2024-04-25T15:23:10.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.263+0000 7f34b820c640 1 -- start start 2024-04-25T15:23:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b820c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34b00bb450 0x7f34b00d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b820c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f34b00d0de0 0x7f34b00b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b820c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b00b5230 con 0x7f34b0163c00 2024-04-25T15:23:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b820c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f34b006c8c0 con 0x7f34b00bb450 2024-04-25T15:23:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b820c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f34b006cba0 con 0x7f34b00d0de0 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b57a7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34b00bb450 0x7f34b00d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b57a7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34b00bb450 0x7f34b00d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37486/0 (socket says 172.21.15.146:37486) 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b57a7640 1 -- 172.21.15.146:0/2835673650 learned_addr learned my addr 172.21.15.146:0/2835673650 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.264+0000 7f34b67a9640 1 --2- 172.21.15.146:0/2835673650 >> v2:172.21.15.177:3300/0 conn(0x7f34b00d0de0 0x7f34b00b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b57a7640 1 -- 172.21.15.146:0/2835673650 >> v2:172.21.15.177:3300/0 conn(0x7f34b00d0de0 msgr2=0x7f34b00b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b57a7640 1 --2- 172.21.15.146:0/2835673650 >> v2:172.21.15.177:3300/0 conn(0x7f34b00d0de0 0x7f34b00b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b57a7640 1 -- 172.21.15.146:0/2835673650 >> v1:172.21.15.177:6789/0 conn(0x7f34b0163c00 legacy=0x7f34b006d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b57a7640 1 -- 172.21.15.146:0/2835673650 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b006ce80 con 0x7f34b00bb450 2024-04-25T15:23:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b67a9640 1 --2- 172.21.15.146:0/2835673650 >> v2:172.21.15.177:3300/0 conn(0x7f34b00d0de0 0x7f34b00b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b57a7640 1 --2- 172.21.15.146:0/2835673650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b00bb450 0x7f34b00d08a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009870 tx=0x7f34a002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d01389802545c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.265+0000 7f34b4fa6640 1 -- 172.21.15.146:0/2835673650 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0037070 con 0x7f34b00bb450 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b4fa6640 1 -- 172.21.15.146:0/2835673650 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a0032440 con 0x7f34b00bb450 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b4fa6640 1 -- 172.21.15.146:0/2835673650 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0032810 con 0x7f34b00bb450 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 -- 172.21.15.146:0/2835673650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b00bb450 msgr2=0x7f34b00d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 --2- 172.21.15.146:0/2835673650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b00bb450 0x7f34b00d08a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009870 tx=0x7f34a002a210 comp rx=0 tx=0).stop 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 -- 172.21.15.146:0/2835673650 shutdown_connections 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 --2- 172.21.15.146:0/2835673650 >> v2:172.21.15.177:3300/0 conn(0x7f34b00d0de0 0x7f34b00b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 --2- 172.21.15.146:0/2835673650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b00bb450 0x7f34b00d08a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 -- 172.21.15.146:0/2835673650 >> 172.21.15.146:0/2835673650 conn(0x7f34b00b7970 msgr2=0x7f34b00b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 -- 172.21.15.146:0/2835673650 shutdown_connections 2024-04-25T15:23:10.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.266+0000 7f34b820c640 1 -- 172.21.15.146:0/2835673650 wait complete. 2024-04-25T15:23:10.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.268+0000 7f34b820c640 1 Processor -- start 2024-04-25T15:23:10.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.268+0000 7f34b820c640 1 -- start start 2024-04-25T15:23:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b820c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34b00bb450 0x7f34b0156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b820c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34b00d0de0 0x7f34b015c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b820c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 0x7f34b0158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b820c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f34b00bb860 con 0x7f34b0163c00 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b820c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34b006c550 con 0x7f34b00d0de0 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b820c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f34b00bb9e0 con 0x7f34b00bb450 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b67a9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 0x7f34b0158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b67a9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 0x7f34b0158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37490/0 (socket says 172.21.15.146:37490) 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.269+0000 7f34b67a9640 1 -- 172.21.15.146:0/794013557 learned_addr learned my addr 172.21.15.146:0/794013557 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b57a7640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34b00d0de0 0x7f34b015c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b5fa8640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34b00bb450 0x7f34b0156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b67a9640 1 -- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34b00bb450 msgr2=0x7f34b0156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b67a9640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34b00bb450 0x7f34b0156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b67a9640 1 -- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34b00d0de0 msgr2=0x7f34b015c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b67a9640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34b00d0de0 0x7f34b015c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b67a9640 1 -- 172.21.15.146:0/794013557 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b015cb10 con 0x7f34b0163c00 2024-04-25T15:23:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b67a9640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 0x7f34b0158060 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f34a800d5b0 tx=0x7f34a800da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8018070 con 0x7f34b0163c00 2024-04-25T15:23:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a8004030 con 0x7f34b0163c00 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b0157830 con 0x7f34b0163c00 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8004400 con 0x7f34b0163c00 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.270+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b015a040 con 0x7f34b0163c00 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.271+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b006a0e0 con 0x7f34b0163c00 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.272+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f34a8014620 con 0x7f34b0163c00 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.272+0000 7f34a67fc640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3494074600 0x7f3494076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.272+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f34a8095790 con 0x7f34b0163c00 2024-04-25T15:23:10.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.272+0000 7f34b5fa8640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3494074600 0x7f3494076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.273+0000 7f34b5fa8640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3494074600 0x7f3494076b10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f34ac001e70 tx=0x7f34ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.277+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34b015cb10 con 0x7f34b0163c00 2024-04-25T15:23:10.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.456+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f34b0160320 con 0x7f34b0163c00 2024-04-25T15:23:10.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.457+0000 7f34a67fc640 1 -- 172.21.15.146:0/794013557 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f34a805f5a0 con 0x7f34b0163c00 2024-04-25T15:23:10.458 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-25T15:23:10.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3494074600 msgr2=0x7f3494076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3494074600 0x7f3494076b10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f34ac001e70 tx=0x7f34ac009040 comp rx=0 tx=0).stop 2024-04-25T15:23:10.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 msgr2=0x7f34b0158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 0x7f34b0158060 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f34a800d5b0 tx=0x7f34a800da70 comp rx=0 tx=0).stop 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 shutdown_connections 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f3494074600 0x7f3494076b10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b0163c00 0x7f34b0158060 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34b00d0de0 0x7f34b015c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 --2- 172.21.15.146:0/794013557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f34b00bb450 0x7f34b0156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 >> 172.21.15.146:0/794013557 conn(0x7f34b00b7970 msgr2=0x7f34b00d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 shutdown_connections 2024-04-25T15:23:10.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.459+0000 7f34b820c640 1 -- 172.21.15.146:0/794013557 wait complete. 2024-04-25T15:23:10.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:23:10.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.582+0000 7faa82934640 1 Processor -- start 2024-04-25T15:23:10.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.582+0000 7faa82934640 1 -- start start 2024-04-25T15:23:10.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa82934640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faa7c0d1140 0x7faa7c0b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa82934640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 0x7faa7c1600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa82934640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7c0b9520 con 0x7faa7c0b4c30 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa82934640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7faa7c06cc00 con 0x7faa7c0d1140 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa82934640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faa7c06c5f0 con 0x7faa7c0b9110 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa7bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7faa7c0b4c30 0x7faa7c0d0a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46766/0 (socket says 172.21.15.146:46766) 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa7bfff640 1 -- 172.21.15.146:0/3821207656 learned_addr learned my addr 172.21.15.146:0/3821207656 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021619641 0 0) 0x7faa7c0b9520 con 0x7faa7c0b4c30 2024-04-25T15:23:10.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa64003520 con 0x7faa7c0b4c30 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa80ed1640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 0x7faa7c1600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.583+0000 7faa7b7fe640 1 --2- 172.21.15.146:0/3821207656 >> v2:172.21.15.177:3300/0 conn(0x7faa7c0d1140 0x7faa7c0b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 340562159 0 0) 0x7faa64003520 con 0x7faa7c0b4c30 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 msgr2=0x7faa7c1600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 0x7faa7c1600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 >> v2:172.21.15.177:3300/0 conn(0x7faa7c0d1140 msgr2=0x7faa7c0b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 --2- 172.21.15.146:0/3821207656 >> v2:172.21.15.177:3300/0 conn(0x7faa7c0d1140 0x7faa7c0b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c1647b0 con 0x7faa7c0b4c30 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7faa6c001670 con 0x7faa7c0b4c30 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 >> v1:172.21.15.146:6789/0 conn(0x7faa7c0b4c30 legacy=0x7faa7c0d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa64004800 0x7faa64004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 0x7faa64005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c06cc00 con 0x7faa640039b0 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c06c5f0 con 0x7faa64004800 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faa6400b9c0 con 0x7faa7c0b9110 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7bfff640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7b7fe640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa64004800 0x7faa64004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa80ed1640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 0x7faa64005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7bfff640 1 -- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 msgr2=0x7faa64005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7bfff640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c0b9110 0x7faa64005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7bfff640 1 -- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa64004800 msgr2=0x7faa64004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7bfff640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa64004800 0x7faa64004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.584+0000 7faa7bfff640 1 -- 172.21.15.146:0/3821207656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa64003520 con 0x7faa640039b0 2024-04-25T15:23:10.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.585+0000 7faa7bfff640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa64003e20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0d24b0 tx=0x7faa64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3942cb80bb729453 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.585+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c007d50 con 0x7faa640039b0 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.585+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6c0058d0 con 0x7faa640039b0 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.585+0000 7faa7affd640 1 -- 172.21.15.146:0/3821207656 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c005d00 con 0x7faa640039b0 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa82934640 1 -- 172.21.15.146:0/3821207656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 msgr2=0x7faa64003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa82934640 1 --2- 172.21.15.146:0/3821207656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa64003e20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0d24b0 tx=0x7faa64005830 comp rx=0 tx=0).stop 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa80ed1640 1 -- 172.21.15.146:0/3821207656 reap_dead start 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa82934640 1 -- 172.21.15.146:0/3821207656 shutdown_connections 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa82934640 1 -- 172.21.15.146:0/3821207656 >> 172.21.15.146:0/3821207656 conn(0x7faa7c0b7970 msgr2=0x7faa7c0b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa82934640 1 -- 172.21.15.146:0/3821207656 shutdown_connections 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.586+0000 7faa82934640 1 -- 172.21.15.146:0/3821207656 wait complete. 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.588+0000 7faa82934640 1 Processor -- start 2024-04-25T15:23:10.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.588+0000 7faa82934640 1 -- start start 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa82934640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa7c15e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa82934640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c1f98b0 0x7faa7c1f9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa82934640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa7c1ebaf0 0x7faa7c1ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa82934640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c0d16b0 con 0x7faa640039b0 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa82934640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c164a90 con 0x7faa7c1ebaf0 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa82934640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7faa7c1600d0 con 0x7faa7c1f98b0 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa7bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa7c15e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa7bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa7c15e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51068/0 (socket says 172.21.15.146:51068) 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa7bfff640 1 -- 172.21.15.146:0/1861973424 learned_addr learned my addr 172.21.15.146:0/1861973424 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa80ed1640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa7c1ebaf0 0x7faa7c1ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.589+0000 7faa7b7fe640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c1f98b0 0x7faa7c1f9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa7bfff640 1 -- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c1f98b0 msgr2=0x7faa7c1f9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa7bfff640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c1f98b0 0x7faa7c1f9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa7bfff640 1 -- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa7c1ebaf0 msgr2=0x7faa7c1ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa7bfff640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa7c1ebaf0 0x7faa7c1ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa7bfff640 1 -- 172.21.15.146:0/1861973424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c0cafe0 con 0x7faa640039b0 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa80ed1640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa7c1ebaf0 0x7faa7c1ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa7bfff640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa7c15e520 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7faa6c02adf0 tx=0x7faa7c164980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c038040 con 0x7faa640039b0 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6c042070 con 0x7faa640039b0 2024-04-25T15:23:10.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7c1ec440 con 0x7faa640039b0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c001c80 con 0x7faa640039b0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.590+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7c0cb250 con 0x7faa640039b0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.591+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa7c06a0e0 con 0x7faa640039b0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.592+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7faa6c007270 con 0x7faa640039b0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.592+0000 7faa78ff9640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa680745d0 0x7faa68076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.592+0000 7faa7b7fe640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa680745d0 0x7faa68076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.592+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7faa6c0b5640 con 0x7faa640039b0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.593+0000 7faa7b7fe640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa680745d0 0x7faa68076ae0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0b6250 tx=0x7faa70009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.597+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa7c1ec440 con 0x7faa640039b0 2024-04-25T15:23:10.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.779+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa7c0d3fa0 con 0x7faa640039b0 2024-04-25T15:23:10.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.780+0000 7faa78ff9640 1 -- 172.21.15.146:0/1861973424 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7faa6c07fa90 con 0x7faa640039b0 2024-04-25T15:23:10.781 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-25T15:23:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa680745d0 msgr2=0x7faa68076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa680745d0 0x7faa68076ae0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0b6250 tx=0x7faa70009190 comp rx=0 tx=0).stop 2024-04-25T15:23:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 msgr2=0x7faa7c15e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa7c15e520 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7faa6c02adf0 tx=0x7faa7c164980 comp rx=0 tx=0).stop 2024-04-25T15:23:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 shutdown_connections 2024-04-25T15:23:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7faa680745d0 0x7faa68076ae0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa7c1ebaf0 0x7faa7c1ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa7c1f98b0 0x7faa7c1f9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 --2- 172.21.15.146:0/1861973424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa640039b0 0x7faa7c15e520 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 >> 172.21.15.146:0/1861973424 conn(0x7faa7c0b7970 msgr2=0x7faa7c0b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 shutdown_connections 2024-04-25T15:23:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.782+0000 7faa82934640 1 -- 172.21.15.146:0/1861973424 wait complete. 2024-04-25T15:23:10.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:23:10.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.903+0000 7f894e405640 1 Processor -- start 2024-04-25T15:23:10.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.903+0000 7f894e405640 1 -- start start 2024-04-25T15:23:10.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.904+0000 7f894e405640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89480cc280 0x7f8948070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.904+0000 7f894e405640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89481618f0 0x7f8948161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.904+0000 7f894e405640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8948162240 con 0x7f894815e300 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.904+0000 7f894e405640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f894815bcd0 con 0x7f89481618f0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.904+0000 7f894e405640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f89480cd490 con 0x7f89480cc280 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89481618f0 0x7f8948161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89481618f0 0x7f8948161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51072/0 (socket says 172.21.15.146:51072) 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 -- 172.21.15.146:0/1746656584 learned_addr learned my addr 172.21.15.146:0/1746656584 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f8947fff640 1 --2- 172.21.15.146:0/1746656584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89480cc280 0x7f8948070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 -- 172.21.15.146:0/1746656584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89480cc280 msgr2=0x7f8948070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 --2- 172.21.15.146:0/1746656584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89480cc280 0x7f8948070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 -- 172.21.15.146:0/1746656584 >> v1:172.21.15.177:6789/0 conn(0x7f894815e300 legacy=0x7f894815e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.905+0000 7f89477fe640 1 -- 172.21.15.146:0/1746656584 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480711d0 con 0x7f89481618f0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.906+0000 7f89477fe640 1 --2- 172.21.15.146:0/1746656584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948161d00 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f893c009870 tx=0x7f893c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef125a137f9a84c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.906+0000 7f8946ffd640 1 -- 172.21.15.146:0/1746656584 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c037070 con 0x7f89481618f0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.906+0000 7f8946ffd640 1 -- 172.21.15.146:0/1746656584 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893c032440 con 0x7f89481618f0 2024-04-25T15:23:10.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.906+0000 7f8946ffd640 1 -- 172.21.15.146:0/1746656584 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c032810 con 0x7f89481618f0 2024-04-25T15:23:10.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 -- 172.21.15.146:0/1746656584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 msgr2=0x7f8948161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 --2- 172.21.15.146:0/1746656584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948161d00 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f893c009870 tx=0x7f893c02a130 comp rx=0 tx=0).stop 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 -- 172.21.15.146:0/1746656584 shutdown_connections 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 --2- 172.21.15.146:0/1746656584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948161d00 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 --2- 172.21.15.146:0/1746656584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89480cc280 0x7f8948070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 -- 172.21.15.146:0/1746656584 >> 172.21.15.146:0/1746656584 conn(0x7f89480ce0c0 msgr2=0x7f89480ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 -- 172.21.15.146:0/1746656584 shutdown_connections 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.907+0000 7f894e405640 1 -- 172.21.15.146:0/1746656584 wait complete. 2024-04-25T15:23:10.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.909+0000 7f894e405640 1 Processor -- start 2024-04-25T15:23:10.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.909+0000 7f894e405640 1 -- start start 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f894e405640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 0x7f8948155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f894e405640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f894815e300 0x7f8948158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f894e405640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f894e405640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89480cce40 con 0x7f89481618f0 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f894e405640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89480ccb60 con 0x7f89480cc280 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f894e405640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f89480cd120 con 0x7f894815e300 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f8947fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 0x7f8948155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f8947fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 0x7f8948155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:52938/0 (socket says 172.21.15.146:52938) 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f8947fff640 1 -- 172.21.15.146:0/3496444545 learned_addr learned my addr 172.21.15.146:0/3496444545 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.911+0000 7f89477fe640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f894815e300 0x7f8948158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f894c9a2640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f8947fff640 1 -- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f894815e300 msgr2=0x7f8948158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f8947fff640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f894815e300 0x7f8948158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f8947fff640 1 -- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 msgr2=0x7f8948155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:10.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f8947fff640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:10.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f8947fff640 1 -- 172.21.15.146:0/3496444545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480b5ba0 con 0x7f89480cc280 2024-04-25T15:23:10.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f89477fe640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f894815e300 0x7f8948158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:10.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.912+0000 7f8947fff640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 0x7f8948155170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f893800e870 tx=0x7f893800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.915+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8938017070 con 0x7f89480cc280 2024-04-25T15:23:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.915+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893800c8e0 con 0x7f89480cc280 2024-04-25T15:23:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.915+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893800ccd0 con 0x7f89480cc280 2024-04-25T15:23:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.915+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89480b5e80 con 0x7f89480cc280 2024-04-25T15:23:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.915+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89480b6440 con 0x7f89480cc280 2024-04-25T15:23:10.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.916+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f894806a0e0 con 0x7f89480cc280 2024-04-25T15:23:10.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.917+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f893801a070 con 0x7f89480cc280 2024-04-25T15:23:10.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.917+0000 7f8944ff9640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:10.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.917+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f8938095740 con 0x7f89480cc280 2024-04-25T15:23:10.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.917+0000 7f89477fe640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:10.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.918+0000 7f89477fe640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f893c034000 tx=0x7f893c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:10.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:10.922+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f893805f550 con 0x7f89480cc280 2024-04-25T15:23:11.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.138+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f894806f8f0 con 0x7f89480cc280 2024-04-25T15:23:11.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.139+0000 7f8944ff9640 1 -- 172.21.15.146:0/3496444545 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5191 (secure 0 0 0) 0x7f8938063850 con 0x7f89480cc280 2024-04-25T15:23:11.140 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:11.140 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:53.856951+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6819,"qdb_cluster":[6819]},"id":3}]} 2024-04-25T15:23:11.140 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 36 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 msgr2=0x7f8934076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f893c034000 tx=0x7f893c02e000 comp rx=0 tx=0).stop 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 msgr2=0x7f8948155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 0x7f8948155170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f893800e870 tx=0x7f893800ed30 comp rx=0 tx=0).stop 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 shutdown_connections 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89481618f0 0x7f8948155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f894815e300 0x7f8948158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 --2- 172.21.15.146:0/3496444545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89480cc280 0x7f8948155170 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 >> 172.21.15.146:0/3496444545 conn(0x7f89480ce0c0 msgr2=0x7f894815f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:11.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 shutdown_connections 2024-04-25T15:23:11.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.141+0000 7f894e405640 1 -- 172.21.15.146:0/3496444545 wait complete. 2024-04-25T15:23:11.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:11.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.262+0000 7f88568f2640 1 Processor -- start 2024-04-25T15:23:11.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.263+0000 7f88568f2640 1 -- start start 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f88568f2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88500cfbe0 0x7f88500cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f88568f2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f88500d19d0 0x7f88500d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f88568f2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500b3920 0x7f88500b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f88568f2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f88500b5020 con 0x7f88500cfbe0 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f88568f2640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f88500b4a60 con 0x7f88500d19d0 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f88568f2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f88500b4d40 con 0x7f88500b3920 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88500cfbe0 0x7f88500cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88500cfbe0 0x7f88500cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51084/0 (socket says 172.21.15.146:51084) 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 -- 172.21.15.146:0/3895360098 learned_addr learned my addr 172.21.15.146:0/3895360098 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884f7fe640 1 --2- 172.21.15.146:0/3895360098 >> v2:172.21.15.177:3300/0 conn(0x7f88500d19d0 0x7f88500d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f8854e8f640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500b3920 0x7f88500b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 -- 172.21.15.146:0/3895360098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500b3920 msgr2=0x7f88500b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500b3920 0x7f88500b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 -- 172.21.15.146:0/3895360098 >> v2:172.21.15.177:3300/0 conn(0x7f88500d19d0 msgr2=0x7f88500d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 --2- 172.21.15.146:0/3895360098 >> v2:172.21.15.177:3300/0 conn(0x7f88500d19d0 0x7f88500d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884ffff640 1 -- 172.21.15.146:0/3895360098 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88500cead0 con 0x7f88500cfbe0 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f8854e8f640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500b3920 0x7f88500b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.264+0000 7f884f7fe640 1 --2- 172.21.15.146:0/3895360098 >> v2:172.21.15.177:3300/0 conn(0x7f88500d19d0 0x7f88500d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.265+0000 7f884ffff640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500cfbe0 0x7f88500cf260 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f88500b8510 tx=0x7f884002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5984a67ec1c881e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.265+0000 7f884effd640 1 -- 172.21.15.146:0/3895360098 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840004030 con 0x7f88500cfbe0 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.265+0000 7f884effd640 1 -- 172.21.15.146:0/3895360098 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88400041d0 con 0x7f88500cfbe0 2024-04-25T15:23:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.265+0000 7f884effd640 1 -- 172.21.15.146:0/3895360098 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840003610 con 0x7f88500cfbe0 2024-04-25T15:23:11.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 -- 172.21.15.146:0/3895360098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500cfbe0 msgr2=0x7f88500cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500cfbe0 0x7f88500cf260 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f88500b8510 tx=0x7f884002a250 comp rx=0 tx=0).stop 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 -- 172.21.15.146:0/3895360098 shutdown_connections 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500b3920 0x7f88500b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 --2- 172.21.15.146:0/3895360098 >> v2:172.21.15.177:3300/0 conn(0x7f88500d19d0 0x7f88500d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 --2- 172.21.15.146:0/3895360098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500cfbe0 0x7f88500cf260 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 -- 172.21.15.146:0/3895360098 >> 172.21.15.146:0/3895360098 conn(0x7f88500b73b0 msgr2=0x7f88500b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 -- 172.21.15.146:0/3895360098 shutdown_connections 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.266+0000 7f88568f2640 1 -- 172.21.15.146:0/3895360098 wait complete. 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.268+0000 7f88568f2640 1 Processor -- start 2024-04-25T15:23:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.268+0000 7f88568f2640 1 -- start start 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.269+0000 7f88568f2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 0x7f885009d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.269+0000 7f88568f2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500cfbe0 0x7f8850071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.269+0000 7f88568f2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88500d19d0 0x7f8850071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f88568f2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f88500cf7a0 con 0x7f88500b3920 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f88568f2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88500d2320 con 0x7f88500d19d0 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f88568f2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f88500c82d0 con 0x7f88500cfbe0 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 0x7f885009d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 0x7f885009d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51100/0 (socket says 172.21.15.146:51100) 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 -- 172.21.15.146:0/2741514459 learned_addr learned my addr 172.21.15.146:0/2741514459 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884f7fe640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500cfbe0 0x7f8850071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f8854e8f640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88500d19d0 0x7f8850071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 -- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500cfbe0 msgr2=0x7f8850071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500cfbe0 0x7f8850071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 -- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88500d19d0 msgr2=0x7f8850071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88500d19d0 0x7f8850071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 -- 172.21.15.146:0/2741514459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88501f6700 con 0x7f88500b3920 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884f7fe640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500cfbe0 0x7f8850071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884ffff640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 0x7f885009d890 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f884002a290 tx=0x7f88400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840041020 con 0x7f88500b3920 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8840003c50 con 0x7f88500b3920 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88501f69e0 con 0x7f88500b3920 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840004030 con 0x7f88500b3920 2024-04-25T15:23:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.270+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885009f4f0 con 0x7f88500b3920 2024-04-25T15:23:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.271+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885006a0e0 con 0x7f88500b3920 2024-04-25T15:23:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.272+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f8840044020 con 0x7f88500b3920 2024-04-25T15:23:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.272+0000 7f884cff9640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f883c0745d0 0x7f883c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.273+0000 7f884f7fe640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f883c0745d0 0x7f883c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.273+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f88400b6340 con 0x7f88500b3920 2024-04-25T15:23:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.273+0000 7f884f7fe640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f883c0745d0 0x7f883c076ae0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f88500722a0 tx=0x7f8844008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.277+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f884004b030 con 0x7f88500b3920 2024-04-25T15:23:11.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.456+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88500cf920 con 0x7f88500b3920 2024-04-25T15:23:11.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.457+0000 7f884cff9640 1 -- 172.21.15.146:0/2741514459 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7f88400807a0 con 0x7f88500b3920 2024-04-25T15:23:11.458 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:11.458 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:11.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f883c0745d0 msgr2=0x7f883c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f883c0745d0 0x7f883c076ae0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f88500722a0 tx=0x7f8844008040 comp rx=0 tx=0).stop 2024-04-25T15:23:11.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 msgr2=0x7f885009d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 0x7f885009d890 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f884002a290 tx=0x7f88400046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 shutdown_connections 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f883c0745d0 0x7f883c076ae0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88500d19d0 0x7f8850071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88500cfbe0 0x7f8850071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 --2- 172.21.15.146:0/2741514459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88500b3920 0x7f885009d890 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 >> 172.21.15.146:0/2741514459 conn(0x7f88500b73b0 msgr2=0x7f88500b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 shutdown_connections 2024-04-25T15:23:11.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.459+0000 7f88568f2640 1 -- 172.21.15.146:0/2741514459 wait complete. 2024-04-25T15:23:11.473 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:23:11.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:23:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.580+0000 7f895731f640 1 Processor -- start 2024-04-25T15:23:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.580+0000 7f895731f640 1 -- start start 2024-04-25T15:23:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.581+0000 7f895731f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f89500cbc90 0x7f8950153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.581+0000 7f895731f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89500cf420 con 0x7f89500d01a0 2024-04-25T15:23:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.581+0000 7f895731f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89500cc270 con 0x7f89500d0ac0 2024-04-25T15:23:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.581+0000 7f895731f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f89501541e0 con 0x7f89500cbc90 2024-04-25T15:23:11.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f89548ba640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f89500d01a0 0x7f895015dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46782/0 (socket says 172.21.15.146:46782) 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f89548ba640 1 -- 172.21.15.146:0/3056764544 learned_addr learned my addr 172.21.15.146:0/3056764544 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f89550bb640 1 --2- 172.21.15.146:0/3056764544 >> v2:172.21.15.177:3300/0 conn(0x7f89500cbc90 0x7f8950153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4071074256 0 0) 0x7f89500cf420 con 0x7f89500d01a0 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f893c003520 con 0x7f89500d01a0 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4047295277 0 0) 0x7f893c003520 con 0x7f89500d01a0 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 >> v2:172.21.15.177:3300/0 conn(0x7f89500cbc90 msgr2=0x7f8950153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 --2- 172.21.15.146:0/3056764544 >> v2:172.21.15.177:3300/0 conn(0x7f89500cbc90 0x7f8950153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 >> v1:172.21.15.177:6789/0 conn(0x7f89500d0ac0 legacy=0x7f89500d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89500cf6c0 con 0x7f89500d01a0 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f8940001670 con 0x7f89500d01a0 2024-04-25T15:23:11.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 >> v1:172.21.15.146:6789/0 conn(0x7f89500d01a0 legacy=0x7f895015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c0039b0 0x7f893c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c004800 0x7f893c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f893c0051c0 0x7f893c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f893c00c130 con 0x7f893c0051c0 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f893c00c2e0 con 0x7f893c0039b0 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f893c00c4e0 con 0x7f893c004800 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f89558bc640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f893c0051c0 0x7f893c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f89550bb640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c0039b0 0x7f893c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.582+0000 7f89548ba640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c004800 0x7f893c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 -- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c004800 msgr2=0x7f893c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c004800 0x7f893c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 -- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c0039b0 msgr2=0x7f893c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f893c0039b0 0x7f893c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 -- 172.21.15.146:0/3056764544 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893c003520 con 0x7f893c0051c0 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 -- 172.21.15.146:0/3056764544 reap_dead start 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f89558bc640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f893c0051c0 0x7f893c00b9c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8948001e20 tx=0x7f893c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70816fa6c1c9307 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948001a70 con 0x7f893c0051c0 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f894802ad70 con 0x7f893c0051c0 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.583+0000 7f8947fff640 1 -- 172.21.15.146:0/3056764544 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948003b90 con 0x7f893c0051c0 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 -- 172.21.15.146:0/3056764544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f893c0051c0 msgr2=0x7f893c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f893c0051c0 0x7f893c00b9c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8948001e20 tx=0x7f893c005830 comp rx=0 tx=0).stop 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 -- 172.21.15.146:0/3056764544 shutdown_connections 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 --2- 172.21.15.146:0/3056764544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f893c0051c0 0x7f893c00b9c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 -- 172.21.15.146:0/3056764544 >> 172.21.15.146:0/3056764544 conn(0x7f89500ce0c0 msgr2=0x7f89500ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 -- 172.21.15.146:0/3056764544 shutdown_connections 2024-04-25T15:23:11.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.584+0000 7f895731f640 1 -- 172.21.15.146:0/3056764544 wait complete. 2024-04-25T15:23:11.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.586+0000 7f895731f640 1 Processor -- start 2024-04-25T15:23:11.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.586+0000 7f895731f640 1 -- start start 2024-04-25T15:23:11.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f895731f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c0051c0 0x7f89501f8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f895731f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 0x7f89500b6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f895731f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895006fc20 0x7f8950070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f895731f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89500d05b0 con 0x7f89501f4a80 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f895731f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89500cfd10 con 0x7f895006fc20 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f895731f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f89500d0730 con 0x7f893c0051c0 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f89548ba640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 0x7f89500b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f89548ba640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 0x7f89500b6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51128/0 (socket says 172.21.15.146:51128) 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f89548ba640 1 -- 172.21.15.146:0/3885644017 learned_addr learned my addr 172.21.15.146:0/3885644017 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f89550bb640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c0051c0 0x7f89501f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.587+0000 7f89558bc640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895006fc20 0x7f8950070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f89548ba640 1 -- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c0051c0 msgr2=0x7f89501f8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f89548ba640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c0051c0 0x7f89501f8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f89548ba640 1 -- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895006fc20 msgr2=0x7f8950070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f89548ba640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895006fc20 0x7f8950070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f89548ba640 1 -- 172.21.15.146:0/3885644017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8950157250 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f89548ba640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 0x7f89500b6760 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f894000e730 tx=0x7f894000eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894000c6c0 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8940018070 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8950070570 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894000cac0 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.588+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8950156500 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.589+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8950156270 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.590+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f894000cc60 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.590+0000 7f8945ffb640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.590+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f89400951c0 con 0x7f89501f4a80 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.590+0000 7f89550bb640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.591+0000 7f89550bb640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f895006c600 tx=0x7f894c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.595+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8950070570 con 0x7f89501f4a80 2024-04-25T15:23:11.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.815+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f895006a0e0 con 0x7f89501f4a80 2024-04-25T15:23:11.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.816+0000 7f8945ffb640 1 -- 172.21.15.146:0/3885644017 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5191 (secure 0 0 0) 0x7f894005f6b0 con 0x7f89501f4a80 2024-04-25T15:23:11.817 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:11.817 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:53.856951+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6819,"qdb_cluster":[6819]},"id":3}]} 2024-04-25T15:23:11.817 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 36 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 msgr2=0x7f8934076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f895006c600 tx=0x7f894c009190 comp rx=0 tx=0).stop 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 msgr2=0x7f89500b6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 0x7f89500b6760 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f894000e730 tx=0x7f894000eb30 comp rx=0 tx=0).stop 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 shutdown_connections 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8934074600 0x7f8934076b10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f895006fc20 0x7f8950070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89501f4a80 0x7f89500b6760 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 --2- 172.21.15.146:0/3885644017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f893c0051c0 0x7f89501f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 >> 172.21.15.146:0/3885644017 conn(0x7f89500ce0c0 msgr2=0x7f89500b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:11.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 shutdown_connections 2024-04-25T15:23:11.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.818+0000 7f895731f640 1 -- 172.21.15.146:0/3885644017 wait complete. 2024-04-25T15:23:11.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:23:11.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.942+0000 7f9dce66e640 1 Processor -- start 2024-04-25T15:23:11.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.942+0000 7f9dce66e640 1 -- start start 2024-04-25T15:23:11.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dce66e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc80d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dce66e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9dc80d0020 0x7f9dc80d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dce66e640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc80b9200 con 0x7f9dc8163d80 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dce66e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9dc806c550 con 0x7f9dc80d0020 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dce66e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc806c830 con 0x7f9dc80d4f60 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9dc80d0020 0x7f9dc80d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9dc80d0020 0x7f9dc80d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51136/0 (socket says 172.21.15.146:51136) 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 -- 172.21.15.146:0/1582482494 learned_addr learned my addr 172.21.15.146:0/1582482494 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc7fff640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc80d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 -- 172.21.15.146:0/1582482494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 msgr2=0x7f9dc80d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc80d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 -- 172.21.15.146:0/1582482494 >> v1:172.21.15.177:6789/0 conn(0x7f9dc8163d80 legacy=0x7f9dc8164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:11.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc77fe640 1 -- 172.21.15.146:0/1582482494 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc80d0970 con 0x7f9dc80d0020 2024-04-25T15:23:11.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.944+0000 7f9dc7fff640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc80d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:11.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.945+0000 7f9dc77fe640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc80d0020 0x7f9dc80d0430 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009870 tx=0x7f9dbc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68782a41b097e6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.945+0000 7f9dc6ffd640 1 -- 172.21.15.146:0/1582482494 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc037070 con 0x7f9dc80d0020 2024-04-25T15:23:11.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.945+0000 7f9dc6ffd640 1 -- 172.21.15.146:0/1582482494 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dbc02ae10 con 0x7f9dc80d0020 2024-04-25T15:23:11.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.945+0000 7f9dc6ffd640 1 -- 172.21.15.146:0/1582482494 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc032660 con 0x7f9dc80d0020 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 -- 172.21.15.146:0/1582482494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc80d0020 msgr2=0x7f9dc80d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc80d0020 0x7f9dc80d0430 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009870 tx=0x7f9dbc02a100 comp rx=0 tx=0).stop 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 -- 172.21.15.146:0/1582482494 shutdown_connections 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc80d0020 0x7f9dc80d0430 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 --2- 172.21.15.146:0/1582482494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc80d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 -- 172.21.15.146:0/1582482494 >> 172.21.15.146:0/1582482494 conn(0x7f9dc80d2450 msgr2=0x7f9dc80d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:11.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 -- 172.21.15.146:0/1582482494 shutdown_connections 2024-04-25T15:23:11.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.946+0000 7f9dce66e640 1 -- 172.21.15.146:0/1582482494 wait complete. 2024-04-25T15:23:11.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.948+0000 7f9dce66e640 1 Processor -- start 2024-04-25T15:23:11.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.948+0000 7f9dce66e640 1 -- start start 2024-04-25T15:23:11.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dce66e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dc80d0020 0x7f9dc8158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dce66e640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc8161690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dce66e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 0x7f9dc8154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dce66e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc80d15f0 con 0x7f9dc8163d80 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dce66e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc80d1070 con 0x7f9dc80d0020 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dce66e640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc80b5150 con 0x7f9dc80d4f60 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.949+0000 7f9dccc0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 0x7f9dc8154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 0x7f9dc8154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51140/0 (socket says 172.21.15.146:51140) 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 -- 172.21.15.146:0/4118150944 learned_addr learned my addr 172.21.15.146:0/4118150944 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dc77fe640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc8161690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dc7fff640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dc80d0020 0x7f9dc8158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 -- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 msgr2=0x7f9dc8161690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc8161690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 -- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dc80d0020 msgr2=0x7f9dc8158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dc80d0020 0x7f9dc8158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 -- 172.21.15.146:0/4118150944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc8161bd0 con 0x7f9dc8163d80 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dccc0b640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 0x7f9dc8154e60 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000be90 tx=0x7f9dc000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0019070 con 0x7f9dc8163d80 2024-04-25T15:23:11.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc0014440 con 0x7f9dc8163d80 2024-04-25T15:23:11.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc815d080 con 0x7f9dc8163d80 2024-04-25T15:23:11.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0014870 con 0x7f9dc8163d80 2024-04-25T15:23:11.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.950+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc815d600 con 0x7f9dc8163d80 2024-04-25T15:23:11.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.952+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc8158a10 con 0x7f9dc8163d80 2024-04-25T15:23:11.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.953+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f9dc001c070 con 0x7f9dc8163d80 2024-04-25T15:23:11.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.953+0000 7f9dc4ff9640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9db40745d0 0x7f9db4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:11.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.953+0000 7f9dc7fff640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9db40745d0 0x7f9db4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:11.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.953+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f9dc0095670 con 0x7f9dc8163d80 2024-04-25T15:23:11.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.954+0000 7f9dc7fff640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9db40745d0 0x7f9db4076ae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80b4880 tx=0x7f9db8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:11.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:11.958+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9dc005f3e0 con 0x7f9dc8163d80 2024-04-25T15:23:12.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.176+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dc806a0e0 con 0x7f9dc8163d80 2024-04-25T15:23:12.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.177+0000 7f9dc4ff9640 1 -- 172.21.15.146:0/4118150944 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5191 (secure 0 0 0) 0x7f9dc00636e0 con 0x7f9dc8163d80 2024-04-25T15:23:12.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:12.178 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:22:19.583041+0000","modified":"2024-04-25T15:22:53.856951+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6819},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6819":{"gid":6819,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/2021557185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2021557185},{"type":"v1","addr":"172.21.15.146:6836","nonce":2021557185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6819,"qdb_cluster":[6819]},"id":3}]} 2024-04-25T15:23:12.178 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 36 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9db40745d0 msgr2=0x7f9db4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9db40745d0 0x7f9db4076ae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80b4880 tx=0x7f9db8008040 comp rx=0 tx=0).stop 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 msgr2=0x7f9dc8154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 0x7f9dc8154e60 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000be90 tx=0x7f9dc000bf80 comp rx=0 tx=0).stop 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 shutdown_connections 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9db40745d0 0x7f9db4076ae0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9dc8163d80 0x7f9dc8154e60 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9dc80d4f60 0x7f9dc8161690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 --2- 172.21.15.146:0/4118150944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dc80d0020 0x7f9dc8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 >> 172.21.15.146:0/4118150944 conn(0x7f9dc80d2450 msgr2=0x7f9dc80b5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:12.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 shutdown_connections 2024-04-25T15:23:12.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.179+0000 7f9dce66e640 1 -- 172.21.15.146:0/4118150944 wait complete. 2024-04-25T15:23:12.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:12.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.298+0000 7fac3ab6c640 1 Processor -- start 2024-04-25T15:23:12.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.299+0000 7fac3ab6c640 1 -- start start 2024-04-25T15:23:12.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac3ab6c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 0x7fac340d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac3ab6c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac340d4a30 con 0x7fac3406d9f0 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac3ab6c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac340d4d10 con 0x7fac340d0750 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac3ab6c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fac3406c550 con 0x7fac34163aa0 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac39109640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fac3406d9f0 0x7fac3415fee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46792/0 (socket says 172.21.15.146:46792) 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac39109640 1 -- 172.21.15.146:0/525402491 learned_addr learned my addr 172.21.15.146:0/525402491 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac38908640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 0x7fac340d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776866978 0 0) 0x7fac340d4a30 con 0x7fac3406d9f0 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.300+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac20003520 con 0x7fac3406d9f0 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3424155499 0 0) 0x7fac20003520 con 0x7fac3406d9f0 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 msgr2=0x7fac340d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 0x7fac340d43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 >> v1:172.21.15.177:6789/0 conn(0x7fac340d0750 legacy=0x7fac340d0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac340d4ff0 con 0x7fac3406d9f0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fac24001670 con 0x7fac3406d9f0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 >> v1:172.21.15.146:6789/0 conn(0x7fac3406d9f0 legacy=0x7fac3415fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac200039e0 0x7fac20003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac20004800 0x7fac20004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 0x7fac20005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac3406c550 con 0x7fac20004800 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac2000ba70 con 0x7fac200039e0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fac2000bc70 con 0x7fac34163aa0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac33fff640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac20004800 0x7fac20004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac38908640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac200039e0 0x7fac20003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac39109640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 0x7fac20005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac33fff640 1 -- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 msgr2=0x7fac20005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac33fff640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34163aa0 0x7fac20005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac33fff640 1 -- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac200039e0 msgr2=0x7fac20003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac33fff640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac200039e0 0x7fac20003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.301+0000 7fac33fff640 1 -- 172.21.15.146:0/525402491 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac20003520 con 0x7fac20004800 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.302+0000 7fac33fff640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac20004800 0x7fac20004c10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fac340d04d0 tx=0x7fac20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752eb3ae4573fc63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.302+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c001460 con 0x7fac20004800 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.302+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1c02ac30 con 0x7fac20004800 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.302+0000 7fac337fe640 1 -- 172.21.15.146:0/525402491 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c007260 con 0x7fac20004800 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac3ab6c640 1 -- 172.21.15.146:0/525402491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac20004800 msgr2=0x7fac20004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/525402491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac20004800 0x7fac20004c10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fac340d04d0 tx=0x7fac20005830 comp rx=0 tx=0).stop 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac39109640 1 -- 172.21.15.146:0/525402491 reap_dead start 2024-04-25T15:23:12.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac3ab6c640 1 -- 172.21.15.146:0/525402491 shutdown_connections 2024-04-25T15:23:12.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac3ab6c640 1 -- 172.21.15.146:0/525402491 >> 172.21.15.146:0/525402491 conn(0x7fac340d6280 msgr2=0x7fac340b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:12.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac3ab6c640 1 -- 172.21.15.146:0/525402491 shutdown_connections 2024-04-25T15:23:12.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.303+0000 7fac3ab6c640 1 -- 172.21.15.146:0/525402491 wait complete. 2024-04-25T15:23:12.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.305+0000 7fac3ab6c640 1 Processor -- start 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.305+0000 7fac3ab6c640 1 -- start start 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac3ab6c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac20004800 0x7fac3415c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac3ab6c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34162570 0x7fac34162980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac3ab6c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 0x7fac341602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac3ab6c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac340d43a0 con 0x7fac3415fee0 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac3ab6c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac340d1580 con 0x7fac20004800 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac3ab6c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fac340d4520 con 0x7fac34162570 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac39109640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 0x7fac341602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac39109640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 0x7fac341602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51148/0 (socket says 172.21.15.146:51148) 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac39109640 1 -- 172.21.15.146:0/2491184569 learned_addr learned my addr 172.21.15.146:0/2491184569 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:12.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac33fff640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34162570 0x7fac34162980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.306+0000 7fac38908640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac20004800 0x7fac3415c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac39109640 1 -- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34162570 msgr2=0x7fac34162980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac39109640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34162570 0x7fac34162980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac39109640 1 -- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac20004800 msgr2=0x7fac3415c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac39109640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac20004800 0x7fac3415c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac39109640 1 -- 172.21.15.146:0/2491184569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac34160830 con 0x7fac3415fee0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac39109640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 0x7fac341602f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fac20004780 tx=0x7fac24007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2401a070 con 0x7fac3415fee0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2400f8b0 con 0x7fac3415fee0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac341f7380 con 0x7fac3415fee0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2400fca0 con 0x7fac3415fee0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.307+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac341f68a0 con 0x7fac3415fee0 2024-04-25T15:23:12.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.308+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3415cf50 con 0x7fac3415fee0 2024-04-25T15:23:12.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.309+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fac2401d070 con 0x7fac3415fee0 2024-04-25T15:23:12.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.309+0000 7fac317fa640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac140745d0 0x7fac14076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.309+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7fac24094ab0 con 0x7fac3415fee0 2024-04-25T15:23:12.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.309+0000 7fac38908640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac140745d0 0x7fac14076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.310+0000 7fac38908640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac140745d0 0x7fac14076ae0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fac340d3e90 tx=0x7fac2c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.314+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac34160830 con 0x7fac3415fee0 2024-04-25T15:23:12.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.490+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac3406a0e0 con 0x7fac3415fee0 2024-04-25T15:23:12.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.491+0000 7fac317fa640 1 -- 172.21.15.146:0/2491184569 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+17158 (secure 0 0 0) 0x7fac2405e8c0 con 0x7fac3415fee0 2024-04-25T15:23:12.492 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:12.492 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":57,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:22:49.831497+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:22:17.219708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T15:22:18.540792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2","mappings":[{"from":0,"to":6}]},{"pgid":"8.2d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:12.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.493+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac140745d0 msgr2=0x7fac14076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.493+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac140745d0 0x7fac14076ae0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fac340d3e90 tx=0x7fac2c00b040 comp rx=0 tx=0).stop 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.493+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 msgr2=0x7fac341602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.493+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 0x7fac341602f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fac20004780 tx=0x7fac24007720 comp rx=0 tx=0).stop 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 shutdown_connections 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fac140745d0 0x7fac14076ae0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac3415fee0 0x7fac341602f0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac34162570 0x7fac34162980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 --2- 172.21.15.146:0/2491184569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac20004800 0x7fac3415c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 >> 172.21.15.146:0/2491184569 conn(0x7fac340d6280 msgr2=0x7fac340c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 shutdown_connections 2024-04-25T15:23:12.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.494+0000 7fac3ab6c640 1 -- 172.21.15.146:0/2491184569 wait complete. 2024-04-25T15:23:12.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:23:12.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.615+0000 7f28622c6640 1 Processor -- start 2024-04-25T15:23:12.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.615+0000 7f28622c6640 1 -- start start 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.616+0000 7f28622c6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0ce020 0x7f285c1622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.616+0000 7f28622c6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f285c0ce8c0 con 0x7f285c15def0 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.616+0000 7f28622c6640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f285c0ceba0 con 0x7f285c0b5240 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.616+0000 7f28622c6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f285c0d1950 con 0x7f285c0ce020 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f285c15def0 0x7f285c0b6980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46804/0 (socket says 172.21.15.146:46804) 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285bfff640 1 -- 172.21.15.146:0/2854740389 learned_addr learned my addr 172.21.15.146:0/2854740389 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285b7fe640 1 --2- 172.21.15.146:0/2854740389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0ce020 0x7f285c1622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285affd640 1 -- 172.21.15.146:0/2854740389 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824299284 0 0) 0x7f285c0ce8c0 con 0x7f285c15def0 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285affd640 1 -- 172.21.15.146:0/2854740389 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2844003520 con 0x7f285c15def0 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285b7fe640 1 -- 172.21.15.146:0/2854740389 >> v1:172.21.15.177:6789/0 conn(0x7f285c0b5240 legacy=0x7f285c0b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285b7fe640 1 -- 172.21.15.146:0/2854740389 >> v1:172.21.15.146:6789/0 conn(0x7f285c15def0 legacy=0x7f285c0b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285affd640 1 -- 172.21.15.146:0/2854740389 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3937632563 0 0) 0x7f2844003520 con 0x7f285c15def0 2024-04-25T15:23:12.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.617+0000 7f285b7fe640 1 -- 172.21.15.146:0/2854740389 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c0cee80 con 0x7f285c0ce020 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.618+0000 7f285b7fe640 1 --2- 172.21.15.146:0/2854740389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0ce020 0x7f285c1622e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28500097d0 tx=0x7f285002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b99b7570342dee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.618+0000 7f285affd640 1 -- 172.21.15.146:0/2854740389 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850037070 con 0x7f285c0ce020 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.618+0000 7f285affd640 1 -- 172.21.15.146:0/2854740389 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285002ac30 con 0x7f285c0ce020 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.618+0000 7f285affd640 1 -- 172.21.15.146:0/2854740389 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c001670 con 0x7f285c0ce020 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 -- 172.21.15.146:0/2854740389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0ce020 msgr2=0x7f285c1622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 --2- 172.21.15.146:0/2854740389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0ce020 0x7f285c1622e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f28500097d0 tx=0x7f285002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 -- 172.21.15.146:0/2854740389 shutdown_connections 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 --2- 172.21.15.146:0/2854740389 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0ce020 0x7f285c1622e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 -- 172.21.15.146:0/2854740389 >> 172.21.15.146:0/2854740389 conn(0x7f285c0d0240 msgr2=0x7f285c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:12.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 -- 172.21.15.146:0/2854740389 shutdown_connections 2024-04-25T15:23:12.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.619+0000 7f28622c6640 1 -- 172.21.15.146:0/2854740389 wait complete. 2024-04-25T15:23:12.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.621+0000 7f28622c6640 1 Processor -- start 2024-04-25T15:23:12.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.621+0000 7f28622c6640 1 -- start start 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f28622c6640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0b5240 0x7f285c0afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f28622c6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285c0ce020 0x7f285c0cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f28622c6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 0x7f285c0b1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f28622c6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f285c0d23a0 con 0x7f285c15def0 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f28622c6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f285c0b7090 con 0x7f285c0ce020 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f28622c6640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f285c0d28c0 con 0x7f285c0b5240 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 0x7f285c0b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 0x7f285c0b1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51158/0 (socket says 172.21.15.146:51158) 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 -- 172.21.15.146:0/2763443364 learned_addr learned my addr 172.21.15.146:0/2763443364 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f285bfff640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0b5240 0x7f285c0afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f285b7fe640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285c0ce020 0x7f285c0cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 -- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0b5240 msgr2=0x7f285c0afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0b5240 0x7f285c0afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 -- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285c0ce020 msgr2=0x7f285c0cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285c0ce020 0x7f285c0cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f2860863640 1 -- 172.21.15.146:0/2763443364 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c0b2600 con 0x7f285c15def0 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.622+0000 7f285bfff640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0b5240 0x7f285c0afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.623+0000 7f2860863640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 0x7f285c0b1890 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f285400c680 tx=0x7f285400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.623+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854003b70 con 0x7f285c15def0 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.623+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2854003d10 con 0x7f285c15def0 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.623+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285c0ccb30 con 0x7f285c15def0 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.623+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854005310 con 0x7f285c15def0 2024-04-25T15:23:12.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.623+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c0cd0f0 con 0x7f285c15def0 2024-04-25T15:23:12.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.624+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285c0cfce0 con 0x7f285c15def0 2024-04-25T15:23:12.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.625+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f28540054b0 con 0x7f285c15def0 2024-04-25T15:23:12.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.625+0000 7f2858ff9640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28480745d0 0x7f2848076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.625+0000 7f285bfff640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28480745d0 0x7f2848076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.625+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6245+0+0 (secure 0 0 0) 0x7f2854095430 con 0x7f285c15def0 2024-04-25T15:23:12.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.626+0000 7f285bfff640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28480745d0 0x7f2848076ae0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f284c001eb0 tx=0x7f284c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.630+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f285c0cfce0 con 0x7f285c15def0 2024-04-25T15:23:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.842+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f285c0b2390 con 0x7f285c15def0 2024-04-25T15:23:12.843 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-04-25T15:23:12.843+0000 7f79638be640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:23:12.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.844+0000 7f2858ff9640 1 -- 172.21.15.146:0/2763443364 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7f285405f8a0 con 0x7f285c15def0 2024-04-25T15:23:12.845 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:23:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.849+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28480745d0 msgr2=0x7f2848076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.849+0000 7f28622c6640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28480745d0 0x7f2848076ae0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f284c001eb0 tx=0x7f284c009040 comp rx=0 tx=0).stop 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.849+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 msgr2=0x7f285c0b1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.849+0000 7f28622c6640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 0x7f285c0b1890 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f285400c680 tx=0x7f285400cb40 comp rx=0 tx=0).stop 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 shutdown_connections 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f28480745d0 0x7f2848076ae0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f285c15def0 0x7f285c0b1890 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f285c0ce020 0x7f285c0cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 --2- 172.21.15.146:0/2763443364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f285c0b5240 0x7f285c0afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 >> 172.21.15.146:0/2763443364 conn(0x7f285c0d0240 msgr2=0x7f285c06def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 shutdown_connections 2024-04-25T15:23:12.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.850+0000 7f28622c6640 1 -- 172.21.15.146:0/2763443364 wait complete. 2024-04-25T15:23:12.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:23:12.918 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-04-25T15:23:12.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.977+0000 7f92bd620640 1 Processor -- start 2024-04-25T15:23:12.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.977+0000 7f92bd620640 1 -- start start 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92bd620640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b45c0 0x7f92b80b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92bd620640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92b80b4f10 0x7f92b80d46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92bd620640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b8163d70 con 0x7f92b815dc60 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92bd620640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f92b80b5320 con 0x7f92b80b4f10 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92bd620640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f92b815e070 con 0x7f92b80b45c0 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92b80b4f10 0x7f92b80d46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92b80b4f10 0x7f92b80d46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51184/0 (socket says 172.21.15.146:51184) 2024-04-25T15:23:12.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 -- 172.21.15.146:0/289401727 learned_addr learned my addr 172.21.15.146:0/289401727 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:12.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b67fc640 1 --2- 172.21.15.146:0/289401727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b45c0 0x7f92b80b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 -- 172.21.15.146:0/289401727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b45c0 msgr2=0x7f92b80b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:12.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 --2- 172.21.15.146:0/289401727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b45c0 0x7f92b80b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 -- 172.21.15.146:0/289401727 >> v1:172.21.15.177:6789/0 conn(0x7f92b815dc60 legacy=0x7f92b81610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.978+0000 7f92b77fe640 1 -- 172.21.15.146:0/289401727 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b8164010 con 0x7f92b80b4f10 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.979+0000 7f92b77fe640 1 --2- 172.21.15.146:0/289401727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b80b4f10 0x7f92b80d46d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009e50 tx=0x7f92b002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7645c47fe686ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.979+0000 7f92b5ffb640 1 -- 172.21.15.146:0/289401727 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0036070 con 0x7f92b80b4f10 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.979+0000 7f92b5ffb640 1 -- 172.21.15.146:0/289401727 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b0031600 con 0x7f92b80b4f10 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.979+0000 7f92b5ffb640 1 -- 172.21.15.146:0/289401727 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b00319d0 con 0x7f92b80b4f10 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 -- 172.21.15.146:0/289401727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b80b4f10 msgr2=0x7f92b80d46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 --2- 172.21.15.146:0/289401727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b80b4f10 0x7f92b80d46d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009e50 tx=0x7f92b002a5f0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 -- 172.21.15.146:0/289401727 shutdown_connections 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 --2- 172.21.15.146:0/289401727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b80b4f10 0x7f92b80d46d0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 --2- 172.21.15.146:0/289401727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b45c0 0x7f92b80b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 -- 172.21.15.146:0/289401727 >> 172.21.15.146:0/289401727 conn(0x7f92b80b7670 msgr2=0x7f92b80b7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 -- 172.21.15.146:0/289401727 shutdown_connections 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.980+0000 7f92bd620640 1 -- 172.21.15.146:0/289401727 wait complete. 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.982+0000 7f92bd620640 1 Processor -- start 2024-04-25T15:23:12.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.982+0000 7f92bd620640 1 -- start start 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92bd620640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b80b45c0 0x7f92b81fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92bd620640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b4f10 0x7f92b81ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92bd620640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 0x7f92b8162330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92bd620640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f92b80d4c10 con 0x7f92b815dc60 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92bd620640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81645d0 con 0x7f92b80b45c0 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92bd620640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f92b80d4d90 con 0x7f92b80b4f10 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92b77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 0x7f92b8162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92b77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 0x7f92b8162330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51192/0 (socket says 172.21.15.146:51192) 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92b77fe640 1 -- 172.21.15.146:0/2328806991 learned_addr learned my addr 172.21.15.146:0/2328806991 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92b67fc640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b4f10 0x7f92b81ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.983+0000 7f92b6ffd640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b80b45c0 0x7f92b81fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92b77fe640 1 -- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b4f10 msgr2=0x7f92b81ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92b77fe640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b4f10 0x7f92b81ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92b77fe640 1 -- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b80b45c0 msgr2=0x7f92b81fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92b77fe640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b80b45c0 0x7f92b81fb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92b77fe640 1 -- 172.21.15.146:0/2328806991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b814fb00 con 0x7f92b815dc60 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92b77fe640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 0x7f92b8162330 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009e50 tx=0x7f92b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0036050 con 0x7f92b815dc60 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b0042070 con 0x7f92b815dc60 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b8071440 con 0x7f92b815dc60 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0009510 con 0x7f92b815dc60 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.984+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b8150250 con 0x7f92b815dc60 2024-04-25T15:23:12.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.985+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b814fcd0 con 0x7f92b815dc60 2024-04-25T15:23:12.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.986+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f92b0007bf0 con 0x7f92b815dc60 2024-04-25T15:23:12.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.986+0000 7f929f7fe640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92a40745d0 0x7f92a4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:12.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.986+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6331+0+0 (secure 0 0 0) 0x7f92b00b63b0 con 0x7f92b815dc60 2024-04-25T15:23:12.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.986+0000 7f92b6ffd640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92a40745d0 0x7f92a4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:12.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.987+0000 7f92b6ffd640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92a40745d0 0x7f92a4076ae0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f92b80b84d0 tx=0x7f92ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:12.991+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92b814fcd0 con 0x7f92b815dc60 2024-04-25T15:23:13.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.205+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f92b806a0e0 con 0x7f92b815dc60 2024-04-25T15:23:13.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.918+0000 7f929f7fe640 1 -- 172.21.15.146:0/2328806991 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7f92b0080060 con 0x7f92b815dc60 2024-04-25T15:23:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92a40745d0 msgr2=0x7f92a4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92a40745d0 0x7f92a4076ae0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f92b80b84d0 tx=0x7f92ac009040 comp rx=0 tx=0).stop 2024-04-25T15:23:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 msgr2=0x7f92b8162330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 0x7f92b8162330 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009e50 tx=0x7f92b0004a40 comp rx=0 tx=0).stop 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 shutdown_connections 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f92a40745d0 0x7f92a4076ae0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b815dc60 0x7f92b8162330 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92b80b4f10 0x7f92b81ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 --2- 172.21.15.146:0/2328806991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b80b45c0 0x7f92b81fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.921+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 >> 172.21.15.146:0/2328806991 conn(0x7f92b80b7670 msgr2=0x7f92b815bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.922+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 shutdown_connections 2024-04-25T15:23:13.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:13.922+0000 7f92bd620640 1 -- 172.21.15.146:0/2328806991 wait complete. 2024-04-25T15:23:13.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:23:14.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.051+0000 7f2a90af9640 1 Processor -- start 2024-04-25T15:23:14.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.051+0000 7f2a90af9640 1 -- start start 2024-04-25T15:23:14.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a90af9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 0x7f2a8c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:14.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a90af9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c160660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a90af9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8c08cb70 con 0x7f2a8c160ba0 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a90af9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8c161380 con 0x7f2a8c08d600 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a90af9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c161660 con 0x7f2a8c06f060 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a8a59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 0x7f2a8c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a89d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a8a59c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 0x7f2a8c06eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53042/0 (socket says 172.21.15.146:53042) 2024-04-25T15:23:14.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a8a59c640 1 -- 172.21.15.146:0/1676981306 learned_addr learned my addr 172.21.15.146:0/1676981306 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a89d9b640 1 -- 172.21.15.146:0/1676981306 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 msgr2=0x7f2a8c06eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a89d9b640 1 --2- 172.21.15.146:0/1676981306 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 0x7f2a8c06eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.053+0000 7f2a89d9b640 1 -- 172.21.15.146:0/1676981306 >> v1:172.21.15.177:6789/0 conn(0x7f2a8c160ba0 legacy=0x7f2a8c0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a89d9b640 1 -- 172.21.15.146:0/1676981306 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c08ce00 con 0x7f2a8c06f060 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a8959a640 1 -- 172.21.15.146:0/1676981306 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874740724 0 0) 0x7f2a8c08cb70 con 0x7f2a8c160ba0 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a8a59c640 1 --2- 172.21.15.146:0/1676981306 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 0x7f2a8c06eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a89d9b640 1 --2- 172.21.15.146:0/1676981306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c160660 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2a800097a0 tx=0x7f2a8002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77aa15d9205a1eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a8959a640 1 -- 172.21.15.146:0/1676981306 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80037070 con 0x7f2a8c06f060 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a8959a640 1 -- 172.21.15.146:0/1676981306 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8002ae50 con 0x7f2a8c06f060 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.054+0000 7f2a8959a640 1 -- 172.21.15.146:0/1676981306 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80032660 con 0x7f2a8c06f060 2024-04-25T15:23:14.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 -- 172.21.15.146:0/1676981306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 msgr2=0x7f2a8c160660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 --2- 172.21.15.146:0/1676981306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c160660 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2a800097a0 tx=0x7f2a8002a100 comp rx=0 tx=0).stop 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 -- 172.21.15.146:0/1676981306 shutdown_connections 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 --2- 172.21.15.146:0/1676981306 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c160660 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 --2- 172.21.15.146:0/1676981306 >> v2:172.21.15.177:3300/0 conn(0x7f2a8c08d600 0x7f2a8c06eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 -- 172.21.15.146:0/1676981306 >> 172.21.15.146:0/1676981306 conn(0x7f2a8c0b52f0 msgr2=0x7f2a8c0b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 -- 172.21.15.146:0/1676981306 shutdown_connections 2024-04-25T15:23:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.055+0000 7f2a90af9640 1 -- 172.21.15.146:0/1676981306 wait complete. 2024-04-25T15:23:14.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.057+0000 7f2a90af9640 1 Processor -- start 2024-04-25T15:23:14.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.057+0000 7f2a90af9640 1 -- start start 2024-04-25T15:23:14.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c1fa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:14.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 0x7f2a8c15c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:14.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c160ba0 0x7f2a8c162e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c161130 con 0x7f2a8c08d600 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c160fb0 con 0x7f2a8c160ba0 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c0b8d80 con 0x7f2a8c06f060 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 0x7f2a8c15c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 0x7f2a8c15c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51198/0 (socket says 172.21.15.146:51198) 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 -- 172.21.15.146:0/3862257668 learned_addr learned my addr 172.21.15.146:0/3862257668 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a8a59c640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c1fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:14.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a8ad9d640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c160ba0 0x7f2a8c162e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 -- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 msgr2=0x7f2a8c1fa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c1fa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 -- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c160ba0 msgr2=0x7f2a8c162e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c160ba0 0x7f2a8c162e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 -- 172.21.15.146:0/3862257668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c0a1360 con 0x7f2a8c08d600 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a89d9b640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 0x7f2a8c15c620 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2a800098c0 tx=0x7f2a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80041030 con 0x7f2a8c08d600 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8002ac50 con 0x7f2a8c08d600 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.059+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8c0cc550 con 0x7f2a8c08d600 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.060+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80037040 con 0x7f2a8c08d600 2024-04-25T15:23:14.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.060+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8c0c9040 con 0x7f2a8c08d600 2024-04-25T15:23:14.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.061+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a8c06a0e0 con 0x7f2a8c08d600 2024-04-25T15:23:14.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.061+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f2a80044070 con 0x7f2a8c08d600 2024-04-25T15:23:14.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.062+0000 7f2a7affd640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a6c0745d0 0x7f2a6c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:14.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.062+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6331+0+0 (secure 0 0 0) 0x7f2a800b5d60 con 0x7f2a8c08d600 2024-04-25T15:23:14.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.062+0000 7f2a8a59c640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a6c0745d0 0x7f2a6c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:14.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.062+0000 7f2a8a59c640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a6c0745d0 0x7f2a6c076ae0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009ce0 tx=0x7f2a7c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:14.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.066+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a800800f0 con 0x7f2a8c08d600 2024-04-25T15:23:14.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.243+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2a8c08ca50 con 0x7f2a8c08d600 2024-04-25T15:23:14.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.929+0000 7f2a7affd640 1 -- 172.21.15.146:0/3862257668 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) ==== 181+0+0 (secure 0 0 0) 0x7f2a8002ae20 con 0x7f2a8c08d600 2024-04-25T15:23:14.930 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:23:14.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.936+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a6c0745d0 msgr2=0x7f2a6c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:14.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.936+0000 7f2a90af9640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a6c0745d0 0x7f2a6c076ae0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009ce0 tx=0x7f2a7c00b040 comp rx=0 tx=0).stop 2024-04-25T15:23:14.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.936+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 msgr2=0x7f2a8c15c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:14.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.936+0000 7f2a90af9640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 0x7f2a8c15c620 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2a800098c0 tx=0x7f2a800046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.937+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 shutdown_connections 2024-04-25T15:23:14.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.937+0000 7f2a90af9640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f2a6c0745d0 0x7f2a6c076ae0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.937+0000 7f2a90af9640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a8c160ba0 0x7f2a8c162e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.937+0000 7f2a90af9640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a8c08d600 0x7f2a8c15c620 secure :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2a800098c0 tx=0x7f2a800046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.937+0000 7f2a90af9640 1 --2- 172.21.15.146:0/3862257668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a8c06f060 0x7f2a8c1fa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:14.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.937+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 >> 172.21.15.146:0/3862257668 conn(0x7f2a8c0b52f0 msgr2=0x7f2a8c0b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:14.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.942+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 shutdown_connections 2024-04-25T15:23:14.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:14.942+0000 7f2a90af9640 1 -- 172.21.15.146:0/3862257668 wait complete. 2024-04-25T15:23:14.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:23:15.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.075+0000 7f07f2251640 1 Processor -- start 2024-04-25T15:23:15.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.075+0000 7f07f2251640 1 -- start start 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07f2251640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07ec0cc2f0 0x7f07ec15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07f2251640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07ec0cc890 con 0x7f07ec0d0a20 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07f2251640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07ec0ccb70 con 0x7f07ec162050 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07f2251640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f07ec15bde0 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eb7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07ec0cc2f0 0x7f07ec15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eb7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07ec0cc2f0 0x7f07ec15b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51214/0 (socket says 172.21.15.146:51214) 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eb7fe640 1 -- 172.21.15.146:0/3173325685 learned_addr learned my addr 172.21.15.146:0/3173325685 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eaffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f07ec0d0a20 0x7f07ec161940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46830/0 (socket says 172.21.15.146:46830) 2024-04-25T15:23:15.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eb7fe640 1 -- 172.21.15.146:0/3173325685 >> v1:172.21.15.177:6789/0 conn(0x7f07ec162050 legacy=0x7f07ec06c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:15.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eb7fe640 1 -- 172.21.15.146:0/3173325685 >> v1:172.21.15.146:6789/0 conn(0x7f07ec0d0a20 legacy=0x7f07ec161940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:15.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07ea7fc640 1 -- 172.21.15.146:0/3173325685 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955427452 0 0) 0x7f07ec0cc890 con 0x7f07ec0d0a20 2024-04-25T15:23:15.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.076+0000 7f07eb7fe640 1 -- 172.21.15.146:0/3173325685 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec0cce50 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.077+0000 7f07eb7fe640 1 --2- 172.21.15.146:0/3173325685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec15b8a0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009870 tx=0x7f07dc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8367bd4911db646f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.077+0000 7f07ea7fc640 1 -- 172.21.15.146:0/3173325685 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc037070 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.077+0000 7f07ea7fc640 1 -- 172.21.15.146:0/3173325685 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07dc032440 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.077+0000 7f07ea7fc640 1 -- 172.21.15.146:0/3173325685 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc032810 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 -- 172.21.15.146:0/3173325685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 msgr2=0x7f07ec15b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 --2- 172.21.15.146:0/3173325685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec15b8a0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009870 tx=0x7f07dc02a2a0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 -- 172.21.15.146:0/3173325685 shutdown_connections 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 --2- 172.21.15.146:0/3173325685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec15b8a0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 -- 172.21.15.146:0/3173325685 >> 172.21.15.146:0/3173325685 conn(0x7f07ec0ce470 msgr2=0x7f07ec0ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 -- 172.21.15.146:0/3173325685 shutdown_connections 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.078+0000 7f07f2251640 1 -- 172.21.15.146:0/3173325685 wait complete. 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.080+0000 7f07f2251640 1 Processor -- start 2024-04-25T15:23:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.080+0000 7f07f2251640 1 -- start start 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07f2251640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec14a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07f2251640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f07ec0d0a20 0x7f07ec15d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07f2251640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07ec0b6d50 0x7f07ec159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07f2251640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0d0fb0 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07f2251640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0d0e30 con 0x7f07ec0b6d50 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07f2251640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f07ec06cf40 con 0x7f07ec0d0a20 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec14a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:15.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec14a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51228/0 (socket says 172.21.15.146:51228) 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.081+0000 7f07eb7fe640 1 -- 172.21.15.146:0/476059402 learned_addr learned my addr 172.21.15.146:0/476059402 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07eb7fe640 1 -- 172.21.15.146:0/476059402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f07ec0d0a20 msgr2=0x7f07ec15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07eb7fe640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f07ec0d0a20 0x7f07ec15d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07eb7fe640 1 -- 172.21.15.146:0/476059402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07ec0b6d50 msgr2=0x7f07ec159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07eb7fe640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07ec0b6d50 0x7f07ec159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07eb7fe640 1 -- 172.21.15.146:0/476059402 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec15c400 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07eb7fe640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec14a200 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009870 tx=0x7f07dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc041070 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07dc03c870 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ec15c6e0 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc037040 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.082+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ec15cca0 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.083+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ec068c30 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.084+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f07dc044070 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.084+0000 7f07d3fff640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f07d8074600 0x7f07d8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.084+0000 7f07eaffd640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f07d8074600 0x7f07d8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.085+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5893+0+0 (secure 0 0 0) 0x7f07dc0b5f60 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.085+0000 7f07eaffd640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f07d8074600 0x7f07d8076b10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f07e0007fd0 tx=0x7f07e0007960 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.089+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07ec068c30 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.266+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f07ec06a0e0 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.935+0000 7f07d3fff640 1 -- 172.21.15.146:0/476059402 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) ==== 169+0+0 (secure 0 0 0) 0x7f07dc0805b0 con 0x7f07ec0cc2f0 2024-04-25T15:23:15.936 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' removed 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.939+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f07d8074600 msgr2=0x7f07d8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.939+0000 7f07f2251640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f07d8074600 0x7f07d8076b10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f07e0007fd0 tx=0x7f07e0007960 comp rx=0 tx=0).stop 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.939+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 msgr2=0x7f07ec14a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.939+0000 7f07f2251640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec14a200 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009870 tx=0x7f07dc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.940+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 shutdown_connections 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.940+0000 7f07f2251640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f07d8074600 0x7f07d8076b10 secure :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f07e0007fd0 tx=0x7f07e0007960 comp rx=0 tx=0).stop 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.940+0000 7f07f2251640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f07ec0b6d50 0x7f07ec159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.940+0000 7f07f2251640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f07ec0d0a20 0x7f07ec15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.940+0000 7f07f2251640 1 --2- 172.21.15.146:0/476059402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07ec0cc2f0 0x7f07ec14a200 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:15.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.940+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 >> 172.21.15.146:0/476059402 conn(0x7f07ec0ce470 msgr2=0x7f07ec0b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:15.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.941+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 shutdown_connections 2024-04-25T15:23:15.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:15.941+0000 7f07f2251640 1 -- 172.21.15.146:0/476059402 wait complete. 2024-04-25T15:23:15.960 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-04-25T15:23:16.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.108+0000 7f5218a4c640 1 Processor -- start 2024-04-25T15:23:16.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.108+0000 7f5218a4c640 1 -- start start 2024-04-25T15:23:16.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5218a4c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 0x7f521406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5218a4c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 0x7f52140d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5218a4c640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f521406dea0 con 0x7f52140d0de0 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5218a4c640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f52140b5230 con 0x7f5214163c00 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5218a4c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f521406c8c0 con 0x7f52140bb450 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5211d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 0x7f52140d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f521259c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 0x7f521406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5212d9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f52140d0de0 0x7f52140b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:59952/0 (socket says 172.21.15.146:59952) 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5211d9b640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 0x7f52140d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:53182/0 (socket says 172.21.15.146:53182) 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f521259c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 0x7f521406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53074/0 (socket says 172.21.15.146:53074) 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.110+0000 7f5212d9d640 1 -- 172.21.15.146:0/1763270684 learned_addr learned my addr 172.21.15.146:0/1763270684 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f521159a640 1 -- 172.21.15.146:0/1763270684 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1344080280 0 0) 0x7f521406dea0 con 0x7f52140d0de0 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f521159a640 1 -- 172.21.15.146:0/1763270684 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51fc003520 con 0x7f52140d0de0 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f5211d9b640 1 -- 172.21.15.146:0/1763270684 >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 msgr2=0x7f521406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f5211d9b640 1 --2- 172.21.15.146:0/1763270684 >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 0x7f521406d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f5211d9b640 1 -- 172.21.15.146:0/1763270684 >> v1:172.21.15.177:6789/0 conn(0x7f52140d0de0 legacy=0x7f52140b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f5211d9b640 1 -- 172.21.15.146:0/1763270684 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521406cba0 con 0x7f52140bb450 2024-04-25T15:23:16.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.111+0000 7f521259c640 1 --2- 172.21.15.146:0/1763270684 >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 0x7f521406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5211d9b640 1 --2- 172.21.15.146:0/1763270684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 0x7f52140d08a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f52080097a0 tx=0x7f520802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a041c84cfc85bd2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f521159a640 1 -- 172.21.15.146:0/1763270684 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208037070 con 0x7f52140bb450 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f521159a640 1 -- 172.21.15.146:0/1763270684 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520802ae50 con 0x7f52140bb450 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f521159a640 1 -- 172.21.15.146:0/1763270684 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208032640 con 0x7f52140bb450 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5218a4c640 1 -- 172.21.15.146:0/1763270684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 msgr2=0x7f52140d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1763270684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 0x7f52140d08a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f52080097a0 tx=0x7f520802a100 comp rx=0 tx=0).stop 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5218a4c640 1 -- 172.21.15.146:0/1763270684 shutdown_connections 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1763270684 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140bb450 0x7f52140d08a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1763270684 >> v2:172.21.15.177:3300/0 conn(0x7f5214163c00 0x7f521406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.112+0000 7f5218a4c640 1 -- 172.21.15.146:0/1763270684 >> 172.21.15.146:0/1763270684 conn(0x7f52140b7970 msgr2=0x7f52140b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.113+0000 7f5218a4c640 1 -- 172.21.15.146:0/1763270684 shutdown_connections 2024-04-25T15:23:16.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.113+0000 7f5218a4c640 1 -- 172.21.15.146:0/1763270684 wait complete. 2024-04-25T15:23:16.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.114+0000 7f5218a4c640 1 Processor -- start 2024-04-25T15:23:16.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.114+0000 7f5218a4c640 1 -- start start 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5218a4c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 0x7f52140acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5218a4c640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140d0de0 0x7f5214156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5218a4c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5214163c00 0x7f52141f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5218a4c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f521406c550 con 0x7f52140bb450 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5218a4c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f521406d160 con 0x7f5214163c00 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5218a4c640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f52140bb860 con 0x7f52140d0de0 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 0x7f52140acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 0x7f52140acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51244/0 (socket says 172.21.15.146:51244) 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 -- 172.21.15.146:0/1599496082 learned_addr learned my addr 172.21.15.146:0/1599496082 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5211d9b640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140d0de0 0x7f5214156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5212d9d640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5214163c00 0x7f52141f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 -- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140d0de0 msgr2=0x7f5214156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140d0de0 0x7f5214156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 -- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5214163c00 msgr2=0x7f52141f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5214163c00 0x7f52141f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f521259c640 1 -- 172.21.15.146:0/1599496082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52140ce530 con 0x7f52140bb450 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.116+0000 7f5211d9b640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140d0de0 0x7f5214156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.117+0000 7f521259c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 0x7f52140acd70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f5204009e60 tx=0x7f520400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.117+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204018020 con 0x7f52140bb450 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.117+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5204014680 con 0x7f52140bb450 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.117+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52140ad2b0 con 0x7f52140bb450 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.117+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204014a50 con 0x7f52140bb450 2024-04-25T15:23:16.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.117+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52140cea10 con 0x7f52140bb450 2024-04-25T15:23:16.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.118+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f521406a0e0 con 0x7f52140bb450 2024-04-25T15:23:16.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.118+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f5204005d10 con 0x7f52140bb450 2024-04-25T15:23:16.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.119+0000 7f5202ffd640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51f40745d0 0x7f51f4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.119+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f5204095f30 con 0x7f52140bb450 2024-04-25T15:23:16.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.119+0000 7f5211d9b640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51f40745d0 0x7f51f4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.119+0000 7f5211d9b640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51f40745d0 0x7f51f4076ae0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5208034000 tx=0x7f520802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.123+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5204022030 con 0x7f52140bb450 2024-04-25T15:23:16.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.352+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) -- 0x7f52140d3ed0 con 0x7f52140bb450 2024-04-25T15:23:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.357+0000 7f5202ffd640 1 -- 172.21.15.146:0/1599496082 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v22) ==== 101+0+0 (secure 0 0 0) 0x7f52140ce530 con 0x7f52140bb450 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51f40745d0 msgr2=0x7f51f4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51f40745d0 0x7f51f4076ae0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5208034000 tx=0x7f520802e000 comp rx=0 tx=0).stop 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 msgr2=0x7f52140acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 0x7f52140acd70 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f5204009e60 tx=0x7f520400c490 comp rx=0 tx=0).stop 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 shutdown_connections 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f51f40745d0 0x7f51f4076ae0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5214163c00 0x7f52141f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52140d0de0 0x7f5214156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 --2- 172.21.15.146:0/1599496082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52140bb450 0x7f52140acd70 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 >> 172.21.15.146:0/1599496082 conn(0x7f52140b7970 msgr2=0x7f521415ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 shutdown_connections 2024-04-25T15:23:16.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.360+0000 7f5218a4c640 1 -- 172.21.15.146:0/1599496082 wait complete. 2024-04-25T15:23:16.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-25T15:23:16.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.484+0000 7fa980914640 1 Processor -- start 2024-04-25T15:23:16.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.484+0000 7fa980914640 1 -- start start 2024-04-25T15:23:16.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa980914640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa97c15e0f0 0x7fa97c0d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa980914640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c0d2370 0x7fa97c0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa980914640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa97c0d2780 con 0x7fa97c0b0fa0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa980914640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa97c0b13b0 con 0x7fa97c15e0f0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa980914640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa97c0aee10 con 0x7fa97c0d2370 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa979d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa97c15e0f0 0x7fa97c0d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa97a59c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa97c0b0fa0 0x7fa97c0b3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46836/0 (socket says 172.21.15.146:46836) 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa97a59c640 1 -- 172.21.15.146:0/2146914533 learned_addr learned my addr 172.21.15.146:0/2146914533 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa979d9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa97c15e0f0 0x7fa97c0d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51250/0 (socket says 172.21.15.146:51250) 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa97ad9d640 1 --2- 172.21.15.146:0/2146914533 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c0d2370 0x7fa97c0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa979d9b640 1 -- 172.21.15.146:0/2146914533 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c0d2370 msgr2=0x7fa97c0afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa979d9b640 1 --2- 172.21.15.146:0/2146914533 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c0d2370 0x7fa97c0afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa979d9b640 1 -- 172.21.15.146:0/2146914533 >> v1:172.21.15.146:6789/0 conn(0x7fa97c0b0fa0 legacy=0x7fa97c0b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa979d9b640 1 -- 172.21.15.146:0/2146914533 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97c0af0b0 con 0x7fa97c15e0f0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.486+0000 7fa97959a640 1 -- 172.21.15.146:0/2146914533 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972025277 0 0) 0x7fa96c001490 con 0x7fa97c0b0fa0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.487+0000 7fa979d9b640 1 --2- 172.21.15.146:0/2146914533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c15e0f0 0x7fa97c0d1e30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa9700097a0 tx=0x7fa97002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb767c3271d681f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.487+0000 7fa97959a640 1 -- 172.21.15.146:0/2146914533 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970037070 con 0x7fa97c15e0f0 2024-04-25T15:23:16.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.487+0000 7fa97959a640 1 -- 172.21.15.146:0/2146914533 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa970032440 con 0x7fa97c15e0f0 2024-04-25T15:23:16.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.487+0000 7fa97959a640 1 -- 172.21.15.146:0/2146914533 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970032810 con 0x7fa97c15e0f0 2024-04-25T15:23:16.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 -- 172.21.15.146:0/2146914533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c15e0f0 msgr2=0x7fa97c0d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 --2- 172.21.15.146:0/2146914533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c15e0f0 0x7fa97c0d1e30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa9700097a0 tx=0x7fa97002a180 comp rx=0 tx=0).stop 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 -- 172.21.15.146:0/2146914533 shutdown_connections 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 --2- 172.21.15.146:0/2146914533 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c0d2370 0x7fa97c0afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 --2- 172.21.15.146:0/2146914533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c15e0f0 0x7fa97c0d1e30 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 -- 172.21.15.146:0/2146914533 >> 172.21.15.146:0/2146914533 conn(0x7fa97c09b2b0 msgr2=0x7fa97c0b6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 -- 172.21.15.146:0/2146914533 shutdown_connections 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.488+0000 7fa980914640 1 -- 172.21.15.146:0/2146914533 wait complete. 2024-04-25T15:23:16.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.490+0000 7fa980914640 1 Processor -- start 2024-04-25T15:23:16.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.490+0000 7fa980914640 1 -- start start 2024-04-25T15:23:16.491 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:16.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa980914640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa97c0b0fa0 0x7fa97c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa980914640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 0x7fa97c156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa980914640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c15e0f0 0x7fa97c156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa980914640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c0af950 con 0x7fa97c0d2370 2024-04-25T15:23:16.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa980914640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c0af670 con 0x7fa97c0b0fa0 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa980914640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fa97c15e500 con 0x7fa97c15e0f0 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 0x7fa97c156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 0x7fa97c156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51252/0 (socket says 172.21.15.146:51252) 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 -- 172.21.15.146:0/1930657304 learned_addr learned my addr 172.21.15.146:0/1930657304 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa97ad9d640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c15e0f0 0x7fa97c156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa97a59c640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa97c0b0fa0 0x7fa97c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 -- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c15e0f0 msgr2=0x7fa97c156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c15e0f0 0x7fa97c156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 -- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa97c0b0fa0 msgr2=0x7fa97c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa97c0b0fa0 0x7fa97c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.491+0000 7fa979d9b640 1 -- 172.21.15.146:0/1930657304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97c09e0d0 con 0x7fa97c0d2370 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.492+0000 7fa979d9b640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 0x7fa97c156350 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa970035040 tx=0x7fa9700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.492+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970042020 con 0x7fa97c0d2370 2024-04-25T15:23:16.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.492+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9700309b0 con 0x7fa97c0d2370 2024-04-25T15:23:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.492+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa97c09e540 con 0x7fa97c0d2370 2024-04-25T15:23:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.492+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970037040 con 0x7fa97c0d2370 2024-04-25T15:23:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.492+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa97c09eb00 con 0x7fa97c0d2370 2024-04-25T15:23:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.493+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa97c09e2a0 con 0x7fa97c0d2370 2024-04-25T15:23:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.494+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fa970045020 con 0x7fa97c0d2370 2024-04-25T15:23:16.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.494+0000 7fa96affd640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa95c0745d0 0x7fa95c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:16.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.494+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fa9700b5cc0 con 0x7fa97c0d2370 2024-04-25T15:23:16.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.494+0000 7fa97a59c640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa95c0745d0 0x7fa95c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.495+0000 7fa97a59c640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa95c0745d0 0x7fa95c076ae0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0012e0 tx=0x7fa96c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.499+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa97c09e2a0 con 0x7fa97c0d2370 2024-04-25T15:23:16.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.677+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7fa97c06a0e0 con 0x7fa97c0d2370 2024-04-25T15:23:16.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.681+0000 7fa96affd640 1 -- 172.21.15.146:0/1930657304 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v23) ==== 92+0+0 (secure 0 0 0) 0x7fa97004c030 con 0x7fa97c0d2370 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa95c0745d0 msgr2=0x7fa95c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa95c0745d0 0x7fa95c076ae0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0012e0 tx=0x7fa96c009040 comp rx=0 tx=0).stop 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 msgr2=0x7fa97c156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 0x7fa97c156350 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa970035040 tx=0x7fa9700046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 shutdown_connections 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fa95c0745d0 0x7fa95c076ae0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa97c15e0f0 0x7fa97c156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa97c0d2370 0x7fa97c156350 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 --2- 172.21.15.146:0/1930657304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa97c0b0fa0 0x7fa97c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 >> 172.21.15.146:0/1930657304 conn(0x7fa97c09b2b0 msgr2=0x7fa97c070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:16.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 shutdown_connections 2024-04-25T15:23:16.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.683+0000 7fa980914640 1 -- 172.21.15.146:0/1930657304 wait complete. 2024-04-25T15:23:16.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_cache_quiesce_splitauth 2024-04-25T15:23:16.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.805+0000 7f5c38868640 1 Processor -- start 2024-04-25T15:23:16.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.805+0000 7f5c38868640 1 -- start start 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c38868640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c340b8f30 0x7f5c340cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c38868640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3408d590 con 0x7f5c340b4ca0 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c38868640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c340d27d0 con 0x7f5c3408dd50 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c38868640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5c340b9340 con 0x7f5c340b8f30 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c32d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c340b8f30 0x7f5c340cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c31d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5c340b4ca0 0x7f5c340ce6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46848/0 (socket says 172.21.15.146:46848) 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c32d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c340b8f30 0x7f5c340cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51256/0 (socket says 172.21.15.146:51256) 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.806+0000 7f5c31d9b640 1 -- 172.21.15.146:0/1927246373 learned_addr learned my addr 172.21.15.146:0/1927246373 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c32d9d640 1 -- 172.21.15.146:0/1927246373 >> v1:172.21.15.177:6789/0 conn(0x7f5c3408dd50 legacy=0x7f5c340d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c32d9d640 1 -- 172.21.15.146:0/1927246373 >> v1:172.21.15.146:6789/0 conn(0x7f5c340b4ca0 legacy=0x7f5c340ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c32d9d640 1 -- 172.21.15.146:0/1927246373 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c340b48b0 con 0x7f5c340b8f30 2024-04-25T15:23:16.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c32d9d640 1 --2- 172.21.15.146:0/1927246373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c340cee00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0097a0 tx=0x7f5c2c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d96ae2968ae56f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c3159a640 1 -- 172.21.15.146:0/1927246373 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c037070 con 0x7f5c340b8f30 2024-04-25T15:23:16.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c3159a640 1 -- 172.21.15.146:0/1927246373 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2c032440 con 0x7f5c340b8f30 2024-04-25T15:23:16.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.807+0000 7f5c3159a640 1 -- 172.21.15.146:0/1927246373 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c032810 con 0x7f5c340b8f30 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 -- 172.21.15.146:0/1927246373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 msgr2=0x7f5c340cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 --2- 172.21.15.146:0/1927246373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c340cee00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0097a0 tx=0x7f5c2c02a230 comp rx=0 tx=0).stop 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 -- 172.21.15.146:0/1927246373 shutdown_connections 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 --2- 172.21.15.146:0/1927246373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c340cee00 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 -- 172.21.15.146:0/1927246373 >> 172.21.15.146:0/1927246373 conn(0x7f5c340b5430 msgr2=0x7f5c340b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 -- 172.21.15.146:0/1927246373 shutdown_connections 2024-04-25T15:23:16.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.808+0000 7f5c38868640 1 -- 172.21.15.146:0/1927246373 wait complete. 2024-04-25T15:23:16.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.810+0000 7f5c38868640 1 Processor -- start 2024-04-25T15:23:16.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.810+0000 7f5c38868640 1 -- start start 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.811+0000 7f5c38868640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c340b4ca0 0x7f5c341d3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.811+0000 7f5c38868640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c341d6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.811+0000 7f5c38868640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c341d6570 0x7f5c341e8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.811+0000 7f5c38868640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5c340d2340 con 0x7f5c340b8f30 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.811+0000 7f5c38868640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3408d8c0 con 0x7f5c341d6570 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.811+0000 7f5c38868640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5c340d24c0 con 0x7f5c340b4ca0 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c341d6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c341d6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51266/0 (socket says 172.21.15.146:51266) 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 -- 172.21.15.146:0/137054478 learned_addr learned my addr 172.21.15.146:0/137054478 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c3259c640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c340b4ca0 0x7f5c341d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c32d9d640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c341d6570 0x7f5c341e8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 -- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c340b4ca0 msgr2=0x7f5c341d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c340b4ca0 0x7f5c341d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 -- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c341d6570 msgr2=0x7f5c341e8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c341d6570 0x7f5c341e8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 -- 172.21.15.146:0/137054478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c341d6f90 con 0x7f5c340b8f30 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c31d9b640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c341d6030 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c430 tx=0x7f5c2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28017070 con 0x7f5c340b8f30 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c28005ce0 con 0x7f5c340b8f30 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28007310 con 0x7f5c340b8f30 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c341d3c90 con 0x7f5c340b8f30 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.812+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c341d40b0 con 0x7f5c340b8f30 2024-04-25T15:23:16.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.813+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c341d6a20 con 0x7f5c340b8f30 2024-04-25T15:23:16.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.814+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f5c2801a070 con 0x7f5c340b8f30 2024-04-25T15:23:16.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.814+0000 7f5c22ffd640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c140745d0 0x7f5c14076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:16.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.814+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f5c280944d0 con 0x7f5c340b8f30 2024-04-25T15:23:16.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.814+0000 7f5c3259c640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c140745d0 0x7f5c14076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:16.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.815+0000 7f5c3259c640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c140745d0 0x7f5c14076ae0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5c340d13f0 tx=0x7f5c24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:16.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:16.819+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c2805e6b0 con 0x7f5c340b8f30 2024-04-25T15:23:17.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.001+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"} v 0) -- 0x7f5c3406a0e0 con 0x7f5c340b8f30 2024-04-25T15:23:17.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.004+0000 7f5c22ffd640 1 -- 172.21.15.146:0/137054478 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"}]=0 v24) ==== 110+0+0 (secure 0 0 0) 0x7f5c280629b0 con 0x7f5c340b8f30 2024-04-25T15:23:17.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c140745d0 msgr2=0x7f5c14076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c140745d0 0x7f5c14076ae0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5c340d13f0 tx=0x7f5c24009040 comp rx=0 tx=0).stop 2024-04-25T15:23:17.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 msgr2=0x7f5c341d6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c341d6030 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c430 tx=0x7f5c2800c8f0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 shutdown_connections 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5c140745d0 0x7f5c14076ae0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c341d6570 0x7f5c341e8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c340b8f30 0x7f5c341d6030 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.006+0000 7f5c38868640 1 --2- 172.21.15.146:0/137054478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c340b4ca0 0x7f5c341d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.007+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 >> 172.21.15.146:0/137054478 conn(0x7f5c340b5430 msgr2=0x7f5c340b6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.007+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 shutdown_connections 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.007+0000 7f5c38868640 1 -- 172.21.15.146:0/137054478 wait complete. 2024-04-25T15:23:17.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_split_bits 2024-04-25T15:23:17.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.128+0000 7f8fa4ff3640 1 Processor -- start 2024-04-25T15:23:17.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.128+0000 7f8fa4ff3640 1 -- start start 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8fa4ff3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fa006dc70 0x7f8fa0161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8fa4ff3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa015e450 con 0x7f8fa0162180 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8fa4ff3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa015e730 con 0x7f8fa0070b20 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8fa4ff3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa015ed80 con 0x7f8fa006dc70 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8f9dd9b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fa006dc70 0x7f8fa0161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8f9ed9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8fa0162180 0x7f8fa0157870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46864/0 (socket says 172.21.15.146:46864) 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8f9ed9d640 1 -- 172.21.15.146:0/3947041981 learned_addr learned my addr 172.21.15.146:0/3947041981 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8f9dd9b640 1 -- 172.21.15.146:0/3947041981 >> v1:172.21.15.177:6789/0 conn(0x7f8fa0070b20 legacy=0x7f8fa006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8f9dd9b640 1 -- 172.21.15.146:0/3947041981 >> v1:172.21.15.146:6789/0 conn(0x7f8fa0162180 legacy=0x7f8fa0157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.130+0000 7f8f9dd9b640 1 -- 172.21.15.146:0/3947041981 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa015ea10 con 0x7f8fa006dc70 2024-04-25T15:23:17.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.131+0000 7f8f9dd9b640 1 --2- 172.21.15.146:0/3947041981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa006dc70 0x7f8fa0161c40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8fa015ef00 tx=0x7f8f9402a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f9ddce8cbe8633 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.131+0000 7f8f9d59a640 1 -- 172.21.15.146:0/3947041981 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94037070 con 0x7f8fa006dc70 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.131+0000 7f8f9d59a640 1 -- 172.21.15.146:0/3947041981 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f94032440 con 0x7f8fa006dc70 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.131+0000 7f8f9d59a640 1 -- 172.21.15.146:0/3947041981 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94032810 con 0x7f8fa006dc70 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/3947041981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa006dc70 msgr2=0x7f8fa0161c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/3947041981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa006dc70 0x7f8fa0161c40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8fa015ef00 tx=0x7f8f9402a160 comp rx=0 tx=0).stop 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/3947041981 shutdown_connections 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/3947041981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa006dc70 0x7f8fa0161c40 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/3947041981 >> 172.21.15.146:0/3947041981 conn(0x7f8fa00d1c10 msgr2=0x7f8fa00733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/3947041981 shutdown_connections 2024-04-25T15:23:17.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.132+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/3947041981 wait complete. 2024-04-25T15:23:17.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.134+0000 7f8fa4ff3640 1 Processor -- start 2024-04-25T15:23:17.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.134+0000 7f8fa4ff3640 1 -- start start 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8fa4ff3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 0x7f8fa0152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8fa4ff3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa0162180 0x7f8fa009e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8fa4ff3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa009e9b0 0x7f8fa0160760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8fa4ff3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa0070f30 con 0x7f8fa0070b20 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8fa4ff3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa015e1f0 con 0x7f8fa0162180 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8fa4ff3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa00710b0 con 0x7f8fa009e9b0 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 0x7f8fa0152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 0x7f8fa0152000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51284/0 (socket says 172.21.15.146:51284) 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 -- 172.21.15.146:0/1559040148 learned_addr learned my addr 172.21.15.146:0/1559040148 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9ed9d640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa009e9b0 0x7f8fa0160760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9dd9b640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa0162180 0x7f8fa009e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 -- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa009e9b0 msgr2=0x7f8fa0160760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa009e9b0 0x7f8fa0160760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 -- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa0162180 msgr2=0x7f8fa009e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa0162180 0x7f8fa009e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.135+0000 7f8f9e59c640 1 -- 172.21.15.146:0/1559040148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa00c71a0 con 0x7f8fa0070b20 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.136+0000 7f8f9e59c640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 0x7f8fa0152000 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000ba50 tx=0x7f8f9000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.136+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90019070 con 0x7f8fa0070b20 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.136+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f9000cc20 con 0x7f8fa0070b20 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.136+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa00c7480 con 0x7f8fa0070b20 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.136+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90014680 con 0x7f8fa0070b20 2024-04-25T15:23:17.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.136+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa00c7a40 con 0x7f8fa0070b20 2024-04-25T15:23:17.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.137+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa006a0e0 con 0x7f8fa0070b20 2024-04-25T15:23:17.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.137+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f8f9001c070 con 0x7f8fa0070b20 2024-04-25T15:23:17.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.138+0000 7f8f8effd640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f800745d0 0x7f8f80076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.138+0000 7f8f9dd9b640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f800745d0 0x7f8f80076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.138+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f8f90094ef0 con 0x7f8fa0070b20 2024-04-25T15:23:17.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.139+0000 7f8f9dd9b640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f800745d0 0x7f8f80076ae0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8f94034000 tx=0x7f8f9402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.143+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f9005f0d0 con 0x7f8fa0070b20 2024-04-25T15:23:17.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.318+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_split_bits"} v 0) -- 0x7f8fa00d0850 con 0x7f8fa0070b20 2024-04-25T15:23:17.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.322+0000 7f8f8effd640 1 -- 172.21.15.146:0/1559040148 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_split_bits"}]=0 v25) ==== 101+0+0 (secure 0 0 0) 0x7f8f900633d0 con 0x7f8fa0070b20 2024-04-25T15:23:17.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f800745d0 msgr2=0x7f8f80076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f800745d0 0x7f8f80076ae0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8f94034000 tx=0x7f8f9402e000 comp rx=0 tx=0).stop 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 msgr2=0x7f8fa0152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 0x7f8fa0152000 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000ba50 tx=0x7f8f9000bf10 comp rx=0 tx=0).stop 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 shutdown_connections 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f8f800745d0 0x7f8f80076ae0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa009e9b0 0x7f8fa0160760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa0162180 0x7f8fa009e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 --2- 172.21.15.146:0/1559040148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa0070b20 0x7f8fa0152000 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 >> 172.21.15.146:0/1559040148 conn(0x7f8fa00d1c10 msgr2=0x7f8fa006d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 shutdown_connections 2024-04-25T15:23:17.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.325+0000 7f8fa4ff3640 1 -- 172.21.15.146:0/1559040148 wait complete. 2024-04-25T15:23:17.337 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_split_size 2024-04-25T15:23:17.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.446+0000 7fdccb98f640 1 Processor -- start 2024-04-25T15:23:17.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.446+0000 7fdccb98f640 1 -- start start 2024-04-25T15:23:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.447+0000 7fdccb98f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdcc409bba0 0x7fdcc40b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.447+0000 7fdccb98f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc40b1960 con 0x7fdcc40ccd40 2024-04-25T15:23:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.447+0000 7fdccb98f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc40b3ce0 con 0x7fdcc40d2d90 2024-04-25T15:23:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.447+0000 7fdccb98f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fdcc40b1c00 con 0x7fdcc409bba0 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.447+0000 7fdcc972b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdcc40ccd40 0x7fdcc40d2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46870/0 (socket says 172.21.15.146:46870) 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcc972b640 1 -- 172.21.15.146:0/857216259 learned_addr learned my addr 172.21.15.146:0/857216259 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133103570 0 0) 0x7fdcc40b1960 con 0x7fdcc40ccd40 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcc9f2c640 1 --2- 172.21.15.146:0/857216259 >> v2:172.21.15.177:3300/0 conn(0x7fdcc409bba0 0x7fdcc40b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcb0003520 con 0x7fdcc40ccd40 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4158337036 0 0) 0x7fdcb0003520 con 0x7fdcc40ccd40 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 >> v2:172.21.15.177:3300/0 conn(0x7fdcc409bba0 msgr2=0x7fdcc40b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 --2- 172.21.15.146:0/857216259 >> v2:172.21.15.177:3300/0 conn(0x7fdcc409bba0 0x7fdcc40b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 >> v1:172.21.15.177:6789/0 conn(0x7fdcc40d2d90 legacy=0x7fdcc40b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc40b1ee0 con 0x7fdcc40ccd40 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fdcc0001810 con 0x7fdcc40ccd40 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 >> v1:172.21.15.146:6789/0 conn(0x7fdcc40ccd40 legacy=0x7fdcc40d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00039b0 0x7fdcb0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb0004800 0x7fdcb0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00051c0 0x7fdcb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc40b1c00 con 0x7fdcb00039b0 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb000c160 con 0x7fdcb0004800 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb000c360 con 0x7fdcb00051c0 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcc972b640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00039b0 0x7fdcb0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcc8f2a640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb0004800 0x7fdcb0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.448+0000 7fdcc9f2c640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00051c0 0x7fdcb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc972b640 1 -- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00051c0 msgr2=0x7fdcb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc972b640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00051c0 0x7fdcb000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc972b640 1 -- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb0004800 msgr2=0x7fdcb0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc972b640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb0004800 0x7fdcb0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc972b640 1 -- 172.21.15.146:0/857216259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb0003520 con 0x7fdcb00039b0 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc9f2c640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00051c0 0x7fdcb000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc9f2c640 1 -- 172.21.15.146:0/857216259 reap_dead start 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcc972b640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00039b0 0x7fdcb0003e20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40d2620 tx=0x7fdcb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e199e3ecf6468ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0007d00 con 0x7fdcb00039b0 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcc002ae50 con 0x7fdcb00039b0 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.449+0000 7fdcbbfff640 1 -- 172.21.15.146:0/857216259 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00074b0 con 0x7fdcb00039b0 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 -- 172.21.15.146:0/857216259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00039b0 msgr2=0x7fdcb0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00039b0 0x7fdcb0003e20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40d2620 tx=0x7fdcb0005830 comp rx=0 tx=0).stop 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 -- 172.21.15.146:0/857216259 shutdown_connections 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 --2- 172.21.15.146:0/857216259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00039b0 0x7fdcb0003e20 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 -- 172.21.15.146:0/857216259 >> 172.21.15.146:0/857216259 conn(0x7fdcc40b7400 msgr2=0x7fdcc40b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 -- 172.21.15.146:0/857216259 shutdown_connections 2024-04-25T15:23:17.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.450+0000 7fdccb98f640 1 -- 172.21.15.146:0/857216259 wait complete. 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.452+0000 7fdccb98f640 1 Processor -- start 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.452+0000 7fdccb98f640 1 -- start start 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdccb98f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb00039b0 0x7fdcc41f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdccb98f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 0x7fdcc4160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdccb98f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcc4153760 0x7fdcc4153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdccb98f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc40b3940 con 0x7fdcc4160560 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdccb98f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc409bfb0 con 0x7fdcb00039b0 2024-04-25T15:23:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdccb98f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc40b3ac0 con 0x7fdcc4153760 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdcc8f2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 0x7fdcc4160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdcc8f2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 0x7fdcc4160970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51300/0 (socket says 172.21.15.146:51300) 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdcc8f2a640 1 -- 172.21.15.146:0/2134687959 learned_addr learned my addr 172.21.15.146:0/2134687959 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdcc972b640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb00039b0 0x7fdcc41f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.453+0000 7fdcc9f2c640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcc4153760 0x7fdcc4153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcc8f2a640 1 -- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcc4153760 msgr2=0x7fdcc4153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcc8f2a640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcc4153760 0x7fdcc4153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcc8f2a640 1 -- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb00039b0 msgr2=0x7fdcc41f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcc8f2a640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb00039b0 0x7fdcc41f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcc8f2a640 1 -- 172.21.15.146:0/2134687959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc41611b0 con 0x7fdcc4160560 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcc8f2a640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 0x7fdcc4160970 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400f5c0 tx=0x7fdcb400fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb40052e0 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb4018070 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc4161490 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4003830 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.454+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc4161a50 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.455+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc406a0e0 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.456+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fdcb40039d0 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.456+0000 7fdcb9ffb640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdca8074600 0x7fdca8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.456+0000 7fdcc972b640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdca8074600 0x7fdca8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.456+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fdcb4094f20 con 0x7fdcc4160560 2024-04-25T15:23:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.457+0000 7fdcc972b640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdca8074600 0x7fdca8076b10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0032000 tx=0x7fdcc0009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.461+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdcb405f070 con 0x7fdcc4160560 2024-04-25T15:23:17.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.639+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_split_size"} v 0) -- 0x7fdcc406f810 con 0x7fdcc4160560 2024-04-25T15:23:17.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.643+0000 7fdcb9ffb640 1 -- 172.21.15.146:0/2134687959 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_split_size"}]=0 v26) ==== 101+0+0 (secure 0 0 0) 0x7fdcb4063370 con 0x7fdcc4160560 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.645+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdca8074600 msgr2=0x7fdca8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.645+0000 7fdccb98f640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdca8074600 0x7fdca8076b10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0032000 tx=0x7fdcc0009030 comp rx=0 tx=0).stop 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 msgr2=0x7fdcc4160970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 0x7fdcc4160970 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400f5c0 tx=0x7fdcb400fa80 comp rx=0 tx=0).stop 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 shutdown_connections 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdca8074600 0x7fdca8076b10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcc4153760 0x7fdcc4153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcc4160560 0x7fdcc4160970 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 --2- 172.21.15.146:0/2134687959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb00039b0 0x7fdcc41f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 >> 172.21.15.146:0/2134687959 conn(0x7fdcc40b7400 msgr2=0x7fdcc40af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 shutdown_connections 2024-04-25T15:23:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.646+0000 7fdccb98f640 1 -- 172.21.15.146:0/2134687959 wait complete. 2024-04-25T15:23:17.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:23:17.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.768+0000 7fdec86ee640 1 Processor -- start 2024-04-25T15:23:17.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.768+0000 7fdec86ee640 1 -- start start 2024-04-25T15:23:17.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec86ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdec00cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec86ee640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdec009bd80 0x7fdec00ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec86ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec00ce920 con 0x7fdec00d2680 2024-04-25T15:23:17.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec86ee640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fdec00cec00 con 0x7fdec009bd80 2024-04-25T15:23:17.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec86ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00ceee0 con 0x7fdec015e070 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec6c8b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdec00d2680 0x7fdec00d2a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46878/0 (socket says 172.21.15.146:46878) 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec6c8b640 1 -- 172.21.15.146:0/1142096229 learned_addr learned my addr 172.21.15.146:0/1142096229 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.769+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2083793131 0 0) 0x7fdec00ce920 con 0x7fdec00d2680 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeac003520 con 0x7fdec00d2680 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec648a640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdec00cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5c89640 1 --2- 172.21.15.146:0/1142096229 >> v2:172.21.15.177:3300/0 conn(0x7fdec009bd80 0x7fdec00ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266182656 0 0) 0x7fdeac003520 con 0x7fdec00d2680 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 msgr2=0x7fdec00cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdec00cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 >> v2:172.21.15.177:3300/0 conn(0x7fdec009bd80 msgr2=0x7fdec00ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 --2- 172.21.15.146:0/1142096229 >> v2:172.21.15.177:3300/0 conn(0x7fdec009bd80 0x7fdec00ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00cf1c0 con 0x7fdec00d2680 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7fdeb80019f0 con 0x7fdec00d2680 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 >> v1:172.21.15.146:6789/0 conn(0x7fdec00d2680 legacy=0x7fdec00d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac0039b0 0x7fdeac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdeac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac004d40 0x7fdeac005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00cec00 con 0x7fdeac004d40 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00ceee0 con 0x7fdeac0039b0 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac00b9c0 con 0x7fdec015e070 2024-04-25T15:23:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec6c8b640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac004d40 0x7fdeac005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec648a640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac0039b0 0x7fdeac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5c89640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdeac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec6c8b640 1 -- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 msgr2=0x7fdeac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec6c8b640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdeac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec6c8b640 1 -- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac0039b0 msgr2=0x7fdeac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec6c8b640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac0039b0 0x7fdeac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec6c8b640 1 -- 172.21.15.146:0/1142096229 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac003520 con 0x7fdeac004d40 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.770+0000 7fdec5c89640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdec015e070 0x7fdeac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.771+0000 7fdec6c8b640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac004d40 0x7fdeac005150 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80013d0 tx=0x7fdeac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df621d16c742a346 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.771+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb800a4e0 con 0x7fdeac004d40 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.771+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb802f070 con 0x7fdeac004d40 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.771+0000 7fdec5488640 1 -- 172.21.15.146:0/1142096229 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb802bc50 con 0x7fdeac004d40 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec86ee640 1 -- 172.21.15.146:0/1142096229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac004d40 msgr2=0x7fdeac005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1142096229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac004d40 0x7fdeac005150 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80013d0 tx=0x7fdeac005830 comp rx=0 tx=0).stop 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec6c8b640 1 -- 172.21.15.146:0/1142096229 reap_dead start 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec86ee640 1 -- 172.21.15.146:0/1142096229 shutdown_connections 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec86ee640 1 -- 172.21.15.146:0/1142096229 >> 172.21.15.146:0/1142096229 conn(0x7fdec00b7400 msgr2=0x7fdec00b7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec86ee640 1 -- 172.21.15.146:0/1142096229 shutdown_connections 2024-04-25T15:23:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.772+0000 7fdec86ee640 1 -- 172.21.15.146:0/1142096229 wait complete. 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.774+0000 7fdec86ee640 1 Processor -- start 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.774+0000 7fdec86ee640 1 -- start start 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec86ee640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdeac004d40 0x7fdec0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec86ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 0x7fdec01f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec86ee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01f43c0 0x7fdec00c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec86ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00cf9c0 con 0x7fdec01f7c40 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec86ee640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdec00ce5a0 con 0x7fdec01f43c0 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec86ee640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00cfb40 con 0x7fdeac004d40 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 0x7fdec01f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 0x7fdec01f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51332/0 (socket says 172.21.15.146:51332) 2024-04-25T15:23:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 -- 172.21.15.146:0/1770517562 learned_addr learned my addr 172.21.15.146:0/1770517562 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 -- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdeac004d40 msgr2=0x7fdec0160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec648a640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdeac004d40 0x7fdec0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec6c8b640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01f43c0 0x7fdec00c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdeac004d40 0x7fdec0160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 -- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01f43c0 msgr2=0x7fdec00c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01f43c0 0x7fdec00c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.775+0000 7fdec5c89640 1 -- 172.21.15.146:0/1770517562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec0160250 con 0x7fdec01f7c40 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.776+0000 7fdec5c89640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 0x7fdec01f3e80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0011570 tx=0x7fdeb0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.776+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0015930 con 0x7fdec01f7c40 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.776+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb0015ad0 con 0x7fdec01f7c40 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.776+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec01604c0 con 0x7fdec01f7c40 2024-04-25T15:23:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.776+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000f460 con 0x7fdec01f7c40 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.776+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec0160a80 con 0x7fdec01f7c40 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.777+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec01f4920 con 0x7fdec01f7c40 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.777+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fdeb000f600 con 0x7fdec01f7c40 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.778+0000 7fdeb6ffd640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdea40745d0 0x7fdea4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.778+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fdeb0095c00 con 0x7fdec01f7c40 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.778+0000 7fdec648a640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdea40745d0 0x7fdea4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.778+0000 7fdec648a640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdea40745d0 0x7fdea4076ae0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009970 tx=0x7fdebc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.783+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdeb009c080 con 0x7fdec01f7c40 2024-04-25T15:23:17.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.960+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fdec006a0e0 con 0x7fdec01f7c40 2024-04-25T15:23:17.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.964+0000 7fdeb6ffd640 1 -- 172.21.15.146:0/1770517562 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fdeb0019030 con 0x7fdec01f7c40 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdea40745d0 msgr2=0x7fdea4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdea40745d0 0x7fdea4076ae0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009970 tx=0x7fdebc008040 comp rx=0 tx=0).stop 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 msgr2=0x7fdec01f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 0x7fdec01f3e80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0011570 tx=0x7fdeb0011a30 comp rx=0 tx=0).stop 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 shutdown_connections 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdea40745d0 0x7fdea4076ae0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdec01f43c0 0x7fdec00c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec01f7c40 0x7fdec01f3e80 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 --2- 172.21.15.146:0/1770517562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdeac004d40 0x7fdec0160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 >> 172.21.15.146:0/1770517562 conn(0x7fdec00b7400 msgr2=0x7fdec0158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 shutdown_connections 2024-04-25T15:23:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:17.966+0000 7fdec86ee640 1 -- 172.21.15.146:0/1770517562 wait complete. 2024-04-25T15:23:17.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_dir_fragment' 2024-04-25T15:23:18.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.087+0000 7fed1757a640 1 Processor -- start 2024-04-25T15:23:18.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.087+0000 7fed1757a640 1 -- start start 2024-04-25T15:23:18.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed1757a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fed10153ca0 0x7fed1015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:18.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed1757a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed100d0ac0 0x7fed100d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:18.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed1757a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed100d1410 con 0x7fed100cbc90 2024-04-25T15:23:18.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed1757a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fed100cfd10 con 0x7fed100d0ac0 2024-04-25T15:23:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed1757a640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fed100cf420 con 0x7fed10153ca0 2024-04-25T15:23:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15b17640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed100d0ac0 0x7fed100d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15316640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fed100cbc90 0x7fed100d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46880/0 (socket says 172.21.15.146:46880) 2024-04-25T15:23:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15b17640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed100d0ac0 0x7fed100d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51342/0 (socket says 172.21.15.146:51342) 2024-04-25T15:23:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15316640 1 -- 172.21.15.146:0/2315566500 learned_addr learned my addr 172.21.15.146:0/2315566500 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed14b15640 1 --2- 172.21.15.146:0/2315566500 >> v2:172.21.15.177:3300/0 conn(0x7fed10153ca0 0x7fed1015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15b17640 1 -- 172.21.15.146:0/2315566500 >> v2:172.21.15.177:3300/0 conn(0x7fed10153ca0 msgr2=0x7fed1015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15b17640 1 --2- 172.21.15.146:0/2315566500 >> v2:172.21.15.177:3300/0 conn(0x7fed10153ca0 0x7fed1015dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15b17640 1 -- 172.21.15.146:0/2315566500 >> v1:172.21.15.146:6789/0 conn(0x7fed100cbc90 legacy=0x7fed100d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed15b17640 1 -- 172.21.15.146:0/2315566500 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed100cc270 con 0x7fed100d0ac0 2024-04-25T15:23:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.089+0000 7fed07fff640 1 -- 172.21.15.146:0/2315566500 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030292102 0 0) 0x7fed0c001490 con 0x7fed100cbc90 2024-04-25T15:23:18.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.090+0000 7fed15b17640 1 --2- 172.21.15.146:0/2315566500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed100d0ac0 0x7fed100d0ed0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fed100b0af0 tx=0x7fed0802c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d3e78b86c112cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:18.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.090+0000 7fed07fff640 1 -- 172.21.15.146:0/2315566500 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08032770 con 0x7fed100d0ac0 2024-04-25T15:23:18.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.090+0000 7fed07fff640 1 -- 172.21.15.146:0/2315566500 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed08038070 con 0x7fed100d0ac0 2024-04-25T15:23:18.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.090+0000 7fed07fff640 1 -- 172.21.15.146:0/2315566500 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08032b30 con 0x7fed100d0ac0 2024-04-25T15:23:18.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 -- 172.21.15.146:0/2315566500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed100d0ac0 msgr2=0x7fed100d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:18.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 --2- 172.21.15.146:0/2315566500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed100d0ac0 0x7fed100d0ed0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fed100b0af0 tx=0x7fed0802c870 comp rx=0 tx=0).stop 2024-04-25T15:23:18.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 -- 172.21.15.146:0/2315566500 shutdown_connections 2024-04-25T15:23:18.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 --2- 172.21.15.146:0/2315566500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed100d0ac0 0x7fed100d0ed0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 --2- 172.21.15.146:0/2315566500 >> v2:172.21.15.177:3300/0 conn(0x7fed10153ca0 0x7fed1015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 -- 172.21.15.146:0/2315566500 >> 172.21.15.146:0/2315566500 conn(0x7fed100ce0c0 msgr2=0x7fed100ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:18.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 -- 172.21.15.146:0/2315566500 shutdown_connections 2024-04-25T15:23:18.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.091+0000 7fed1757a640 1 -- 172.21.15.146:0/2315566500 wait complete. 2024-04-25T15:23:18.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.093+0000 7fed1757a640 1 Processor -- start 2024-04-25T15:23:18.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.093+0000 7fed1757a640 1 -- start start 2024-04-25T15:23:18.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed1757a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed100cbc90 0x7fed1014e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:18.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed1757a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100d0ac0 0x7fed100aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:18.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed1757a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 0x7fed1006f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed1757a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fed1015e1a0 con 0x7fed10153ca0 2024-04-25T15:23:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed1757a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed100cf9a0 con 0x7fed100d0ac0 2024-04-25T15:23:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed1757a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fed101540b0 con 0x7fed100cbc90 2024-04-25T15:23:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 0x7fed1006f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 0x7fed1006f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51346/0 (socket says 172.21.15.146:51346) 2024-04-25T15:23:18.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 -- 172.21.15.146:0/1686450621 learned_addr learned my addr 172.21.15.146:0/1686450621 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:18.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed14b15640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100d0ac0 0x7fed100aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:18.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15316640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed100cbc90 0x7fed1014e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:18.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 -- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed100cbc90 msgr2=0x7fed1014e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:18.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed100cbc90 0x7fed1014e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 -- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100d0ac0 msgr2=0x7fed100aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:18.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100d0ac0 0x7fed100aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed15b17640 1 -- 172.21.15.146:0/1686450621 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed10154890 con 0x7fed10153ca0 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.094+0000 7fed14b15640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100d0ac0 0x7fed100aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.095+0000 7fed15b17640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 0x7fed1006f380 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fed08035040 tx=0x7fed080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.095+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08004230 con 0x7fed10153ca0 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.095+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed080043d0 con 0x7fed10153ca0 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.095+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed101f5cf0 con 0x7fed10153ca0 2024-04-25T15:23:18.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.095+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08003830 con 0x7fed10153ca0 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.095+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed10154320 con 0x7fed10153ca0 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.096+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed101f5ae0 con 0x7fed10153ca0 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.096+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fed08032070 con 0x7fed10153ca0 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.097+0000 7fed05ffb640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecf40745d0 0x7fecf4076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.097+0000 7fed15316640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecf40745d0 0x7fecf4076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.097+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fed080b80c0 con 0x7fed10153ca0 2024-04-25T15:23:18.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.097+0000 7fed15316640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecf40745d0 0x7fecf4076ae0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0012e0 tx=0x7fed0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:18.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.101+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed08042070 con 0x7fed10153ca0 2024-04-25T15:23:18.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.345+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_dir_fragment"]} v 0) -- 0x7fed1006a0e0 con 0x7fed10153ca0 2024-04-25T15:23:18.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.943+0000 7fed05ffb640 1 -- 172.21.15.146:0/1686450621 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_dir_fragment"]}]=0 v219) ==== 142+0+0 (secure 0 0 0) 0x7fed080827e0 con 0x7fed10153ca0 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecf40745d0 msgr2=0x7fecf4076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecf40745d0 0x7fecf4076ae0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0012e0 tx=0x7fed0c009040 comp rx=0 tx=0).stop 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 msgr2=0x7fed1006f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 0x7fed1006f380 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fed08035040 tx=0x7fed080046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 shutdown_connections 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fecf40745d0 0x7fecf4076ae0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10153ca0 0x7fed1006f380 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100d0ac0 0x7fed100aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 --2- 172.21.15.146:0/1686450621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed100cbc90 0x7fed1014e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 >> 172.21.15.146:0/1686450621 conn(0x7fed100ce0c0 msgr2=0x7fed1014f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:18.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 shutdown_connections 2024-04-25T15:23:18.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:18.945+0000 7fed1757a640 1 -- 172.21.15.146:0/1686450621 wait complete. 2024-04-25T15:23:18.959 INFO:tasks.cephfs_test_runner:That quiesce completes with fragmentation in the background. ... ERROR 2024-04-25T15:23:18.959 INFO:tasks.cephfs_test_runner: 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:ERROR: test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:That quiesce completes with fragmentation in the background. 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs/test_quiesce.py", line 576, in test_quiesce_dir_fragment 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner: reqid = self.reqid_tostr(J['op']['reqid']) 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:AttributeError: 'TestQuiesce' object has no attribute 'reqid_tostr' 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner: 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:Ran 2 tests in 164.398s 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner: 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-04-25T15:23:18.960 INFO:tasks.cephfs_test_runner: 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner:ERROR: test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner:That quiesce completes with fragmentation in the background. 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs/test_quiesce.py", line 576, in test_quiesce_dir_fragment 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner: reqid = self.reqid_tostr(J['op']['reqid']) 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner:AttributeError: 'TestQuiesce' object has no attribute 'reqid_tostr' 2024-04-25T15:23:18.961 INFO:tasks.cephfs_test_runner: 2024-04-25T15:23:18.961 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-25T15:23:19.241 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=af2f4d86f1cb44569d03d0535c036ed9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-25T15:23:19.243 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-25T15:23:19.253 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T15:23:19.265 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T15:23:19.265 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:23:19.265 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:19.285 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1449 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:23:19.286 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-25T15:23:19.286 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:19.343 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1449 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:23:19.344 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:23:19.344 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:19.363 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3659 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:23:19.363 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-25T15:23:19.363 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:23:19.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056067,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3659 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:23:19.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.421 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-04-25T15:23:19.477 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:23:19.478 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:23:19.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:23:19.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:23:19.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.611 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link delete ceph-brx 2024-04-25T15:23:19.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.717 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-25T15:23:19.736 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:23:19.736 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:23:19.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:23:19.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.805 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:23:19.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:23:19.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-04-25T15:23:19.967 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T15:23:19.981 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:23:19.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:23:20.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.092+0000 7fdcb49c5640 1 Processor -- start 2024-04-25T15:23:20.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.092+0000 7fdcb49c5640 1 -- start start 2024-04-25T15:23:20.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.093+0000 7fdcb49c5640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdcb006da10 0x7fdcb00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.093+0000 7fdcb49c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c6cc0 0x7fdcb0164bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.093+0000 7fdcb49c5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb0160410 con 0x7fdcb00c68b0 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcb49c5640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fdcb006c8c0 con 0x7fdcb006da10 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcb49c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb00b2d50 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcadd9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdcb00c68b0 0x7fdcb006cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46884/0 (socket says 172.21.15.146:46884) 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcadd9b640 1 -- 172.21.15.146:0/1068991805 learned_addr learned my addr 172.21.15.146:0/1068991805 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcaed9d640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c6cc0 0x7fdcb0164bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcae59c640 1 --2- 172.21.15.146:0/1068991805 >> v2:172.21.15.177:3300/0 conn(0x7fdcb006da10 0x7fdcb00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcad59a640 1 -- 172.21.15.146:0/1068991805 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388970400 0 0) 0x7fdcb0160410 con 0x7fdcb00c68b0 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcad59a640 1 -- 172.21.15.146:0/1068991805 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc98003520 con 0x7fdcb00c68b0 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcae59c640 1 -- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c6cc0 msgr2=0x7fdcb0164bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcad59a640 1 -- 172.21.15.146:0/1068991805 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1193742490 0 0) 0x7fdc98003520 con 0x7fdcb00c68b0 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcae59c640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c6cc0 0x7fdcb0164bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcae59c640 1 -- 172.21.15.146:0/1068991805 >> v1:172.21.15.146:6789/0 conn(0x7fdcb00c68b0 legacy=0x7fdcb006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcae59c640 1 -- 172.21.15.146:0/1068991805 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb00b3030 con 0x7fdcb006da10 2024-04-25T15:23:20.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.094+0000 7fdcaed9d640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c6cc0 0x7fdcb0164bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:20.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.095+0000 7fdcae59c640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb00d2220 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdca00097a0 tx=0x7fdca002a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecedd4f1b0bae779 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.095+0000 7fdcad59a640 1 -- 172.21.15.146:0/1068991805 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0037070 con 0x7fdcb006da10 2024-04-25T15:23:20.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.095+0000 7fdcad59a640 1 -- 172.21.15.146:0/1068991805 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca002ab10 con 0x7fdcb006da10 2024-04-25T15:23:20.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.095+0000 7fdcad59a640 1 -- 172.21.15.146:0/1068991805 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4001690 con 0x7fdcb006da10 2024-04-25T15:23:20.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 -- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 msgr2=0x7fdcb00d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb00d2220 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdca00097a0 tx=0x7fdca002a080 comp rx=0 tx=0).stop 2024-04-25T15:23:20.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 -- 172.21.15.146:0/1068991805 shutdown_connections 2024-04-25T15:23:20.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c6cc0 0x7fdcb0164bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/1068991805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb00d2220 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 -- 172.21.15.146:0/1068991805 >> 172.21.15.146:0/1068991805 conn(0x7fdcb009b2b0 msgr2=0x7fdcb00b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:20.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 -- 172.21.15.146:0/1068991805 shutdown_connections 2024-04-25T15:23:20.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.096+0000 7fdcb49c5640 1 -- 172.21.15.146:0/1068991805 wait complete. 2024-04-25T15:23:20.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.098+0000 7fdcb49c5640 1 Processor -- start 2024-04-25T15:23:20.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.098+0000 7fdcb49c5640 1 -- start start 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcb49c5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb014e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcb49c5640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c68b0 0x7fdcb014d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcb49c5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 0x7fdcb014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcb49c5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb00d2760 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcb49c5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb006c550 con 0x7fdcb006da10 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcb49c5640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb006d460 con 0x7fdcb00c68b0 2024-04-25T15:23:20.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcaed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 0x7fdcb014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcaed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 0x7fdcb014b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51348/0 (socket says 172.21.15.146:51348) 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcaed9d640 1 -- 172.21.15.146:0/4239259778 learned_addr learned my addr 172.21.15.146:0/4239259778 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcae59c640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.099+0000 7fdcadd9b640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c68b0 0x7fdcb014d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcaed9d640 1 -- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c68b0 msgr2=0x7fdcb014d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcaed9d640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c68b0 0x7fdcb014d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcaed9d640 1 -- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 msgr2=0x7fdcb014e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcaed9d640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb014e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcaed9d640 1 -- 172.21.15.146:0/4239259778 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb014daa0 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcaed9d640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 0x7fdcb014b3b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009e10 tx=0x7fdca800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8018020 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca8014480 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb014c4c0 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8014850 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.100+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb014ca80 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.101+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb006a0e0 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.102+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fdca8005d10 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.102+0000 7fdc9effd640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc90074600 0x7fdc90076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.102+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fdca8095f10 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.102+0000 7fdcae59c640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc90074600 0x7fdc90076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.103+0000 7fdcae59c640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc90074600 0x7fdc90076b10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdca0034000 tx=0x7fdca002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.107+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdca8022030 con 0x7fdcb00c6cc0 2024-04-25T15:23:20.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.280+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdcb00b7130 con 0x7fdc90074600 2024-04-25T15:23:20.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.280+0000 7fdc9effd640 1 -- 172.21.15.146:0/4239259778 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13510 (secure 0 0 0) 0x7fdcb00b7130 con 0x7fdc90074600 2024-04-25T15:23:20.281 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:20.282 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:23:20.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc90074600 msgr2=0x7fdc90076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc90074600 0x7fdc90076b10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdca0034000 tx=0x7fdca002e000 comp rx=0 tx=0).stop 2024-04-25T15:23:20.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 msgr2=0x7fdcb014b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 0x7fdcb014b3b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009e10 tx=0x7fdca800c490 comp rx=0 tx=0).stop 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 shutdown_connections 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fdc90074600 0x7fdc90076b10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcb00c6cc0 0x7fdcb014b3b0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdcb00c68b0 0x7fdcb014d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 --2- 172.21.15.146:0/4239259778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdcb006da10 0x7fdcb014e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 >> 172.21.15.146:0/4239259778 conn(0x7fdcb009b2b0 msgr2=0x7fdcb00684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:20.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 shutdown_connections 2024-04-25T15:23:20.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.282+0000 7fdcb49c5640 1 -- 172.21.15.146:0/4239259778 wait complete. 2024-04-25T15:23:20.294 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-25T15:23:20.032428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297880,"kb_used_data":6824,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754676840,"statfs":{"total":773094113280,"available":772789084160,"internally_reserved":0,"allocated":6987776,"data_stored":3810784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":297913906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1649455644,"num_objects":-5291,"num_object_clones":0,"num_object_copies":-10582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5291,"num_whiteouts":0,"num_read":-83,"num_read_kb":-6532,"num_write":-9719,"num_write_kb":-1625884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-682,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.186448"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":186,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-25T15:23:16.243667+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:23:16.243667+0000","last_peered":"2024-04-25T15:23:16.243667+0000","last_clean":"2024-04-25T15:23:16.243667+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:23:16.243667+0000","last_undegraded":"2024-04-25T15:23:16.243667+0000","last_fullsized":"2024-04-25T15:23:16.243667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40212,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331628,"statfs":{"total":96636764160,"available":96595587072,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40208,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:23:20.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:23:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.402+0000 7f4d4f805640 1 Processor -- start 2024-04-25T15:23:20.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.402+0000 7f4d4f805640 1 -- start start 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4f805640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480d1cc0 0x7f4d480d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4f805640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d48161b30 0x7f4d48161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4f805640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d480d1420 con 0x7f4d480cf1d0 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4f805640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f4d480ced40 con 0x7f4d48161b30 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4f805640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4d480d1700 con 0x7f4d480d1cc0 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d48161b30 0x7f4d48161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4cda0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480d1cc0 0x7f4d480d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d48161b30 0x7f4d48161f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53198/0 (socket says 172.21.15.146:53198) 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4cda0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480d1cc0 0x7f4d480d20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:53300/0 (socket says 172.21.15.146:53300) 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 -- 172.21.15.146:0/266222236 learned_addr learned my addr 172.21.15.146:0/266222236 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 -- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480d1cc0 msgr2=0x7f4d480d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 --2- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480d1cc0 0x7f4d480d20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 -- 172.21.15.146:0/266222236 >> v1:172.21.15.177:6789/0 conn(0x7f4d480cf1d0 legacy=0x7f4d480b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d3ffff640 1 -- 172.21.15.146:0/266222236 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2966187120 0 0) 0x7f4d480d1420 con 0x7f4d480cf1d0 2024-04-25T15:23:20.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.404+0000 7f4d4dda2640 1 -- 172.21.15.146:0/266222236 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d480d19e0 con 0x7f4d48161b30 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.405+0000 7f4d4dda2640 1 --2- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d48161f40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009e50 tx=0x7f4d4002a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23a66235725a6bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.405+0000 7f4d3ffff640 1 -- 172.21.15.146:0/266222236 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40036070 con 0x7f4d48161b30 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.405+0000 7f4d3ffff640 1 -- 172.21.15.146:0/266222236 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d40031600 con 0x7f4d48161b30 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.405+0000 7f4d3ffff640 1 -- 172.21.15.146:0/266222236 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d400319d0 con 0x7f4d48161b30 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 -- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 msgr2=0x7f4d48161f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 --2- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d48161f40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009e50 tx=0x7f4d4002a710 comp rx=0 tx=0).stop 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 -- 172.21.15.146:0/266222236 shutdown_connections 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 --2- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d48161f40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 --2- 172.21.15.146:0/266222236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480d1cc0 0x7f4d480d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 -- 172.21.15.146:0/266222236 >> 172.21.15.146:0/266222236 conn(0x7f4d4809b2b0 msgr2=0x7f4d480b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 -- 172.21.15.146:0/266222236 shutdown_connections 2024-04-25T15:23:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.406+0000 7f4d4f805640 1 -- 172.21.15.146:0/266222236 wait complete. 2024-04-25T15:23:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.408+0000 7f4d4f805640 1 Processor -- start 2024-04-25T15:23:20.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.408+0000 7f4d4f805640 1 -- start start 2024-04-25T15:23:20.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4f805640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480cf1d0 0x7f4d48150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4f805640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 0x7f4d4815f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4f805640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d481f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4f805640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4d480d2610 con 0x7f4d480d1cc0 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4f805640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4d480cf5e0 con 0x7f4d48161b30 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4f805640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4815e310 con 0x7f4d480cf1d0 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4cda0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 0x7f4d4815f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4cda0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 0x7f4d4815f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51354/0 (socket says 172.21.15.146:51354) 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4cda0640 1 -- 172.21.15.146:0/4025737802 learned_addr learned my addr 172.21.15.146:0/4025737802 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4d5a1640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480cf1d0 0x7f4d48150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.409+0000 7f4d4dda2640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d481f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4cda0640 1 -- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480cf1d0 msgr2=0x7f4d48150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4cda0640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480cf1d0 0x7f4d48150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4cda0640 1 -- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 msgr2=0x7f4d481f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4cda0640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d481f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4cda0640 1 -- 172.21.15.146:0/4025737802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4815fb00 con 0x7f4d480d1cc0 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4cda0640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 0x7f4d4815f040 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e870 tx=0x7f4d3800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38017070 con 0x7f4d480d1cc0 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3800c940 con 0x7f4d480d1cc0 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4815fde0 con 0x7f4d480d1cc0 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800cd30 con 0x7f4d480d1cc0 2024-04-25T15:23:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.410+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d481603a0 con 0x7f4d480d1cc0 2024-04-25T15:23:20.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.411+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d4806a0e0 con 0x7f4d480d1cc0 2024-04-25T15:23:20.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.412+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4d3801a070 con 0x7f4d480d1cc0 2024-04-25T15:23:20.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.412+0000 7f4d3dffb640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d2c0745d0 0x7f4d2c076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.413+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f4d38095080 con 0x7f4d480d1cc0 2024-04-25T15:23:20.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.413+0000 7f4d4d5a1640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d2c0745d0 0x7f4d2c076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.413+0000 7f4d4d5a1640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d2c0745d0 0x7f4d2c076ae0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4d44000f60 tx=0x7f4d44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.417+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d3805f260 con 0x7f4d480d1cc0 2024-04-25T15:23:20.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.590+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d480d2a70 con 0x7f4d2c0745d0 2024-04-25T15:23:20.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.591+0000 7f4d3dffb640 1 -- 172.21.15.146:0/4025737802 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13510 (secure 0 0 0) 0x7f4d480d2a70 con 0x7f4d2c0745d0 2024-04-25T15:23:20.592 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:20.592 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:23:20.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d2c0745d0 msgr2=0x7f4d2c076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d2c0745d0 0x7f4d2c076ae0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4d44000f60 tx=0x7f4d44009040 comp rx=0 tx=0).stop 2024-04-25T15:23:20.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 msgr2=0x7f4d4815f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 0x7f4d4815f040 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e870 tx=0x7f4d3800ed30 comp rx=0 tx=0).stop 2024-04-25T15:23:20.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 shutdown_connections 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4d2c0745d0 0x7f4d2c076ae0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d48161b30 0x7f4d481f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d480d1cc0 0x7f4d4815f040 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 --2- 172.21.15.146:0/4025737802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d480cf1d0 0x7f4d48150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 >> 172.21.15.146:0/4025737802 conn(0x7f4d4809b2b0 msgr2=0x7f4d480b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 shutdown_connections 2024-04-25T15:23:20.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.593+0000 7f4d4f805640 1 -- 172.21.15.146:0/4025737802 wait complete. 2024-04-25T15:23:20.605 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-25T15:23:20.032428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297880,"kb_used_data":6824,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754676840,"statfs":{"total":773094113280,"available":772789084160,"internally_reserved":0,"allocated":6987776,"data_stored":3810784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":297913906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1649455644,"num_objects":-5291,"num_object_clones":0,"num_object_copies":-10582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5291,"num_whiteouts":0,"num_read":-83,"num_read_kb":-6532,"num_write":-9719,"num_write_kb":-1625884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-682,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.186448"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":186,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-25T15:23:16.243667+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:23:16.243667+0000","last_peered":"2024-04-25T15:23:16.243667+0000","last_clean":"2024-04-25T15:23:16.243667+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:23:16.243667+0000","last_undegraded":"2024-04-25T15:23:16.243667+0000","last_fullsized":"2024-04-25T15:23:16.243667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40212,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331628,"statfs":{"total":96636764160,"available":96595587072,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40208,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:23:20.605 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:23:20.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:23:20.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.712+0000 7fb9c9a70640 1 Processor -- start 2024-04-25T15:23:20.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.712+0000 7fb9c9a70640 1 -- start start 2024-04-25T15:23:20.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c9a70640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40715e0 0x7fb9c406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c9a70640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9c4070190 0x7fb9c4169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c9a70640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c40705a0 con 0x7fb9c40b7350 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c9a70640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c4071bc0 con 0x7fb9c4070190 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c9a70640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c40d1b00 con 0x7fb9c40715e0 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9c4070190 0x7fb9c4169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb9c4070190 0x7fb9c4169ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56926/0 (socket says 172.21.15.146:56926) 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c3fff640 1 -- 172.21.15.146:0/2450627848 learned_addr learned my addr 172.21.15.146:0/2450627848 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.713+0000 7fb9c2ffd640 1 --2- 172.21.15.146:0/2450627848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40715e0 0x7fb9c406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c3fff640 1 -- 172.21.15.146:0/2450627848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40715e0 msgr2=0x7fb9c406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c3fff640 1 --2- 172.21.15.146:0/2450627848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40715e0 0x7fb9c406fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c3fff640 1 -- 172.21.15.146:0/2450627848 >> v1:172.21.15.177:6789/0 conn(0x7fb9c40b7350 legacy=0x7fb9c4160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c3fff640 1 -- 172.21.15.146:0/2450627848 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c40d1de0 con 0x7fb9c4070190 2024-04-25T15:23:20.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c3fff640 1 --2- 172.21.15.146:0/2450627848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c4169ca0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0098a0 tx=0x7fb9bc02a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbe1ef076cd7903 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c27fc640 1 -- 172.21.15.146:0/2450627848 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc037070 con 0x7fb9c4070190 2024-04-25T15:23:20.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.714+0000 7fb9c27fc640 1 -- 172.21.15.146:0/2450627848 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9bc032440 con 0x7fb9c4070190 2024-04-25T15:23:20.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c27fc640 1 -- 172.21.15.146:0/2450627848 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc032810 con 0x7fb9c4070190 2024-04-25T15:23:20.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 -- 172.21.15.146:0/2450627848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 msgr2=0x7fb9c4169ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/2450627848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c4169ca0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0098a0 tx=0x7fb9bc02a290 comp rx=0 tx=0).stop 2024-04-25T15:23:20.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 -- 172.21.15.146:0/2450627848 shutdown_connections 2024-04-25T15:23:20.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/2450627848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c4169ca0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/2450627848 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40715e0 0x7fb9c406fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 -- 172.21.15.146:0/2450627848 >> 172.21.15.146:0/2450627848 conn(0x7fb9c409b2b0 msgr2=0x7fb9c40b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:20.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 -- 172.21.15.146:0/2450627848 shutdown_connections 2024-04-25T15:23:20.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.715+0000 7fb9c9a70640 1 -- 172.21.15.146:0/2450627848 wait complete. 2024-04-25T15:23:20.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.717+0000 7fb9c9a70640 1 Processor -- start 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.717+0000 7fb9c9a70640 1 -- start start 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.718+0000 7fb9c9a70640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c41513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.718+0000 7fb9c9a70640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c40715e0 0x7fb9c40b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.718+0000 7fb9c9a70640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40b7350 0x7fb9c40b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.718+0000 7fb9c9a70640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c40cf4b0 con 0x7fb9c4070190 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.718+0000 7fb9c9a70640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c40d23a0 con 0x7fb9c40715e0 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.718+0000 7fb9c9a70640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c4070980 con 0x7fb9c40b7350 2024-04-25T15:23:20.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c41513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c41513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56928/0 (socket says 172.21.15.146:56928) 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 -- 172.21.15.146:0/1945593643 learned_addr learned my addr 172.21.15.146:0/1945593643 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c3fff640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40b7350 0x7fb9c40b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c2ffd640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c40715e0 0x7fb9c40b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 -- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40b7350 msgr2=0x7fb9c40b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40b7350 0x7fb9c40b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 -- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c40715e0 msgr2=0x7fb9c40b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c40715e0 0x7fb9c40b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 -- 172.21.15.146:0/1945593643 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c41f6550 con 0x7fb9c4070190 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c3fff640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40b7350 0x7fb9c40b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c2ffd640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c40715e0 0x7fb9c40b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c37fe640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c41513d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c6e0 tx=0x7fb9b400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4016020 con 0x7fb9c4070190 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b4006d50 con 0x7fb9c4070190 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c41f6810 con 0x7fb9c4070190 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4003bb0 con 0x7fb9c4070190 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.719+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c41f6dd0 con 0x7fb9c4070190 2024-04-25T15:23:20.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.720+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c406a0e0 con 0x7fb9c4070190 2024-04-25T15:23:20.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.721+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fb9b40050d0 con 0x7fb9c4070190 2024-04-25T15:23:20.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.721+0000 7fb9bbfff640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb9a4074600 0x7fb9a4076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:20.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.722+0000 7fb9c2ffd640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb9a4074600 0x7fb9a4076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:20.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.722+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fb9b4094e30 con 0x7fb9c4070190 2024-04-25T15:23:20.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.722+0000 7fb9c2ffd640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb9a4074600 0x7fb9a4076b10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb9c406dee0 tx=0x7fb9ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:20.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.726+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9b4020020 con 0x7fb9c4070190 2024-04-25T15:23:20.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.895+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9c40711b0 con 0x7fb9a4074600 2024-04-25T15:23:20.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.896+0000 7fb9bbfff640 1 -- 172.21.15.146:0/1945593643 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13510 (secure 0 0 0) 0x7fb9c40711b0 con 0x7fb9a4074600 2024-04-25T15:23:20.897 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:20.897 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:23:20.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb9a4074600 msgr2=0x7fb9a4076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb9a4074600 0x7fb9a4076b10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb9c406dee0 tx=0x7fb9ac008040 comp rx=0 tx=0).stop 2024-04-25T15:23:20.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 msgr2=0x7fb9c41513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:20.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c41513d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400c6e0 tx=0x7fb9b400cba0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 shutdown_connections 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb9a4074600 0x7fb9a4076b10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c40b7350 0x7fb9c40b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c40715e0 0x7fb9c40b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 --2- 172.21.15.146:0/1945593643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c4070190 0x7fb9c41513d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 >> 172.21.15.146:0/1945593643 conn(0x7fb9c409b2b0 msgr2=0x7fb9c40b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 shutdown_connections 2024-04-25T15:23:20.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:20.898+0000 7fb9c9a70640 1 -- 172.21.15.146:0/1945593643 wait complete. 2024-04-25T15:23:20.910 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-25T15:23:20.032428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":69,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297880,"kb_used_data":6824,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754676840,"statfs":{"total":773094113280,"available":772789084160,"internally_reserved":0,"allocated":6987776,"data_stored":3810784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":297913906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1649455644,"num_objects":-5291,"num_object_clones":0,"num_object_copies":-10582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5291,"num_whiteouts":0,"num_read":-83,"num_read_kb":-6532,"num_write":-9719,"num_write_kb":-1625884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-682,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.186448"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":186,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-25T15:23:16.243667+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:23:16.243667+0000","last_peered":"2024-04-25T15:23:16.243667+0000","last_clean":"2024-04-25T15:23:16.243667+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:23:16.243667+0000","last_undegraded":"2024-04-25T15:23:16.243667+0000","last_fullsized":"2024-04-25T15:23:16.243667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40212,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331628,"statfs":{"total":96636764160,"available":96595587072,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40208,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:23:20.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:23:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.020+0000 7fc828484640 1 Processor -- start 2024-04-25T15:23:21.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.020+0000 7fc828484640 1 -- start start 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc828484640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc8200d4f60 0x7fc8200d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc828484640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc8200d0020 0x7fc8200d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc828484640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8200b9200 con 0x7fc820163d80 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc828484640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc82006c550 con 0x7fc8200d4f60 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc828484640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc82006c830 con 0x7fc8200d0020 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc826220640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc8200d4f60 0x7fc8200d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc826220640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc8200d4f60 0x7fc8200d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56942/0 (socket says 172.21.15.146:56942) 2024-04-25T15:23:21.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc826220640 1 -- 172.21.15.146:0/3493132011 learned_addr learned my addr 172.21.15.146:0/3493132011 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc825a1f640 1 --2- 172.21.15.146:0/3493132011 >> v2:172.21.15.177:3300/0 conn(0x7fc8200d0020 0x7fc8200d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc826220640 1 -- 172.21.15.146:0/3493132011 >> v2:172.21.15.177:3300/0 conn(0x7fc8200d0020 msgr2=0x7fc8200d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc826220640 1 --2- 172.21.15.146:0/3493132011 >> v2:172.21.15.177:3300/0 conn(0x7fc8200d0020 0x7fc8200d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.021+0000 7fc826220640 1 -- 172.21.15.146:0/3493132011 >> v1:172.21.15.177:6789/0 conn(0x7fc820163d80 legacy=0x7fc820164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.022+0000 7fc826220640 1 -- 172.21.15.146:0/3493132011 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8200d0970 con 0x7fc8200d4f60 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.022+0000 7fc825a1f640 1 --2- 172.21.15.146:0/3493132011 >> v2:172.21.15.177:3300/0 conn(0x7fc8200d0020 0x7fc8200d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.022+0000 7fc826220640 1 --2- 172.21.15.146:0/3493132011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc8200d5370 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc8200b8c90 tx=0x7fc81c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922b3a2edf474010 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.022+0000 7fc82521e640 1 -- 172.21.15.146:0/3493132011 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c037070 con 0x7fc8200d4f60 2024-04-25T15:23:21.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.022+0000 7fc82521e640 1 -- 172.21.15.146:0/3493132011 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81c02ae20 con 0x7fc8200d4f60 2024-04-25T15:23:21.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.022+0000 7fc82521e640 1 -- 172.21.15.146:0/3493132011 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c032660 con 0x7fc8200d4f60 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 -- 172.21.15.146:0/3493132011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 msgr2=0x7fc8200d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 --2- 172.21.15.146:0/3493132011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc8200d5370 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc8200b8c90 tx=0x7fc81c02a110 comp rx=0 tx=0).stop 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 -- 172.21.15.146:0/3493132011 shutdown_connections 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 --2- 172.21.15.146:0/3493132011 >> v2:172.21.15.177:3300/0 conn(0x7fc8200d0020 0x7fc8200d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 --2- 172.21.15.146:0/3493132011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc8200d5370 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 -- 172.21.15.146:0/3493132011 >> 172.21.15.146:0/3493132011 conn(0x7fc8200d2450 msgr2=0x7fc8200d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:21.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 -- 172.21.15.146:0/3493132011 shutdown_connections 2024-04-25T15:23:21.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.023+0000 7fc828484640 1 -- 172.21.15.146:0/3493132011 wait complete. 2024-04-25T15:23:21.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.025+0000 7fc828484640 1 Processor -- start 2024-04-25T15:23:21.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.025+0000 7fc828484640 1 -- start start 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc828484640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8200d0020 0x7fc820155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc828484640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc82015f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc828484640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc820163d80 0x7fc820162060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc828484640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc8200d15f0 con 0x7fc8200d4f60 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc828484640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8200d1070 con 0x7fc8200d0020 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc828484640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc8200b5150 con 0x7fc820163d80 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc825a1f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc82015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:21.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc825a1f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc82015f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56958/0 (socket says 172.21.15.146:56958) 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc825a1f640 1 -- 172.21.15.146:0/2903831891 learned_addr learned my addr 172.21.15.146:0/2903831891 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc826220640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8200d0020 0x7fc820155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.026+0000 7fc826a21640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc820163d80 0x7fc820162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc825a1f640 1 -- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc820163d80 msgr2=0x7fc820162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc825a1f640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc820163d80 0x7fc820162060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc825a1f640 1 -- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8200d0020 msgr2=0x7fc820155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc825a1f640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8200d0020 0x7fc820155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc825a1f640 1 -- 172.21.15.146:0/2903831891 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8201f6890 con 0x7fc8200d4f60 2024-04-25T15:23:21.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc825a1f640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc82015f250 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc81000c7f0 tx=0x7fc81000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810015070 con 0x7fc8200d4f60 2024-04-25T15:23:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc810007820 con 0x7fc8200d4f60 2024-04-25T15:23:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810007bd0 con 0x7fc8200d4f60 2024-04-25T15:23:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8201f6b70 con 0x7fc8200d4f60 2024-04-25T15:23:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.027+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8201f7130 con 0x7fc8200d4f60 2024-04-25T15:23:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.028+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82006a0e0 con 0x7fc8200d4f60 2024-04-25T15:23:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.029+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fc810018070 con 0x7fc8200d4f60 2024-04-25T15:23:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.029+0000 7fc816ffd640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc804074600 0x7fc804076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.029+0000 7fc826220640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc804074600 0x7fc804076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.029+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fc810095370 con 0x7fc8200d4f60 2024-04-25T15:23:21.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.030+0000 7fc826220640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc804074600 0x7fc804076b10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc81c034000 tx=0x7fc81c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:21.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.034+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc81005f4d0 con 0x7fc8200d4f60 2024-04-25T15:23:21.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.208+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8200d4100 con 0x7fc8200d4f60 2024-04-25T15:23:21.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.209+0000 7fc816ffd640 1 -- 172.21.15.146:0/2903831891 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+13473 (secure 0 0 0) 0x7fc8100637d0 con 0x7fc8200d4f60 2024-04-25T15:23:21.210 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:21.210 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":60,"fsid":"c6e9a678-3722-48f4-a297-70f509c13e29","created":"2024-04-25T15:19:21.871390+0000","modified":"2024-04-25T15:23:15.930548+0000","last_up_change":"2024-04-25T15:19:32.754401+0000","last_in_change":"2024-04-25T15:19:24.124526+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:19:34.001962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f99c8f48-3894-494a-9239-7f00941858ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6817","nonce":1105656680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6819","nonce":1105656680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6823","nonce":1105656680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1105656680},{"type":"v1","addr":"172.21.15.146:6821","nonce":1105656680}]},"public_addr":"172.21.15.146:6817/1105656680","cluster_addr":"172.21.15.146:6819/1105656680","heartbeat_back_addr":"172.21.15.146:6823/1105656680","heartbeat_front_addr":"172.21.15.146:6821/1105656680","state":["exists","up"]},{"osd":1,"uuid":"13de8f11-bab2-4388-a48e-d1ef86f9ac75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6825","nonce":3545155054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6827","nonce":3545155054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6831","nonce":3545155054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3545155054},{"type":"v1","addr":"172.21.15.146:6829","nonce":3545155054}]},"public_addr":"172.21.15.146:6825/3545155054","cluster_addr":"172.21.15.146:6827/3545155054","heartbeat_back_addr":"172.21.15.146:6831/3545155054","heartbeat_front_addr":"172.21.15.146:6829/3545155054","state":["exists","up"]},{"osd":2,"uuid":"86c182b9-706e-4b35-807b-94719fd66e36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6801","nonce":3921051373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6803","nonce":3921051373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6807","nonce":3921051373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3921051373},{"type":"v1","addr":"172.21.15.146:6805","nonce":3921051373}]},"public_addr":"172.21.15.146:6801/3921051373","cluster_addr":"172.21.15.146:6803/3921051373","heartbeat_back_addr":"172.21.15.146:6807/3921051373","heartbeat_front_addr":"172.21.15.146:6805/3921051373","state":["exists","up"]},{"osd":3,"uuid":"abe109f1-9591-4742-9d25-d7595f02c506","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6809","nonce":2287149109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6811","nonce":2287149109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6815","nonce":2287149109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2287149109},{"type":"v1","addr":"172.21.15.146:6813","nonce":2287149109}]},"public_addr":"172.21.15.146:6809/2287149109","cluster_addr":"172.21.15.146:6811/2287149109","heartbeat_back_addr":"172.21.15.146:6815/2287149109","heartbeat_front_addr":"172.21.15.146:6813/2287149109","state":["exists","up"]},{"osd":4,"uuid":"5bff8dde-d43c-4e63-9e32-d38b5cd88226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6817","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6819","nonce":761443217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6821","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6823","nonce":761443217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6829","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6831","nonce":761443217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6825","nonce":761443217},{"type":"v1","addr":"172.21.15.177:6827","nonce":761443217}]},"public_addr":"172.21.15.177:6819/761443217","cluster_addr":"172.21.15.177:6823/761443217","heartbeat_back_addr":"172.21.15.177:6831/761443217","heartbeat_front_addr":"172.21.15.177:6827/761443217","state":["exists","up"]},{"osd":5,"uuid":"e00f58da-2f23-448c-ac44-7673c0ab01e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6809","nonce":2540256124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6811","nonce":2540256124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6815","nonce":2540256124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":2540256124},{"type":"v1","addr":"172.21.15.177:6813","nonce":2540256124}]},"public_addr":"172.21.15.177:6809/2540256124","cluster_addr":"172.21.15.177:6811/2540256124","heartbeat_back_addr":"172.21.15.177:6815/2540256124","heartbeat_front_addr":"172.21.15.177:6813/2540256124","state":["exists","up"]},{"osd":6,"uuid":"2348a56d-1908-4fae-9b45-53aaf09fc42a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6801","nonce":2938680267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6803","nonce":2938680267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6807","nonce":2938680267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2938680267},{"type":"v1","addr":"172.21.15.177:6805","nonce":2938680267}]},"public_addr":"172.21.15.177:6801/2938680267","cluster_addr":"172.21.15.177:6803/2938680267","heartbeat_back_addr":"172.21.15.177:6807/2938680267","heartbeat_front_addr":"172.21.15.177:6805/2938680267","state":["exists","up"]},{"osd":7,"uuid":"339377f9-bb6b-4cd9-b892-2b0f7bb03df5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6818","nonce":1513283347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6822","nonce":1513283347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6830","nonce":1513283347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1513283347},{"type":"v1","addr":"172.21.15.177:6826","nonce":1513283347}]},"public_addr":"172.21.15.177:6818/1513283347","cluster_addr":"172.21.15.177:6822/1513283347","heartbeat_back_addr":"172.21.15.177:6830/1513283347","heartbeat_front_addr":"172.21.15.177:6826/1513283347","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.352104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.318450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.354612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.308560+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.306432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.232378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.235111+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:19:29.326087+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/2021557185":"2024-04-26T15:23:12.843584+0000","172.21.15.146:6834/2021557185":"2024-04-26T15:23:12.843584+0000","172.21.15.146:6836/865508848":"2024-04-26T15:22:49.831455+0000","172.21.15.146:6834/865508848":"2024-04-26T15:22:49.831455+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:23:21.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc804074600 msgr2=0x7fc804076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:21.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc804074600 0x7fc804076b10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc81c034000 tx=0x7fc81c02e000 comp rx=0 tx=0).stop 2024-04-25T15:23:21.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 msgr2=0x7fc82015f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:21.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc82015f250 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc81000c7f0 tx=0x7fc81000ccb0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 shutdown_connections 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc804074600 0x7fc804076b10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc820163d80 0x7fc820162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8200d4f60 0x7fc82015f250 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 --2- 172.21.15.146:0/2903831891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8200d0020 0x7fc820155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 >> 172.21.15.146:0/2903831891 conn(0x7fc8200d2450 msgr2=0x7fc82006c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 shutdown_connections 2024-04-25T15:23:21.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:21.211+0000 7fc828484640 1 -- 172.21.15.146:0/2903831891 wait complete. 2024-04-25T15:23:22.224 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T15:23:22.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:22.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.350+0000 7f42834f2640 1 Processor -- start 2024-04-25T15:23:22.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.350+0000 7f42834f2640 1 -- start start 2024-04-25T15:23:22.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f42834f2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f427c08d600 0x7f427c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f42834f2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f42834f2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427c08cb70 con 0x7f427c06f060 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f42834f2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f427c161380 con 0x7f427c08d600 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f42834f2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f427c161660 con 0x7f427c160d70 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f428128e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f427c08d600 0x7f427c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f4280a8d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f427c06f060 0x7f427c160660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43524/0 (socket says 172.21.15.146:43524) 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f4280a8d640 1 -- 172.21.15.146:0/1677087103 learned_addr learned my addr 172.21.15.146:0/1677087103 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:22.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f4281a8f640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f428128e640 1 -- 172.21.15.146:0/1677087103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 msgr2=0x7f427c0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f428128e640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c0d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f428128e640 1 -- 172.21.15.146:0/1677087103 >> v1:172.21.15.146:6789/0 conn(0x7f427c06f060 legacy=0x7f427c160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f4273fff640 1 -- 172.21.15.146:0/1677087103 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470173531 0 0) 0x7f427c08cb70 con 0x7f427c06f060 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f428128e640 1 -- 172.21.15.146:0/1677087103 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c08ce00 con 0x7f427c08d600 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.352+0000 7f4281a8f640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c0d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.353+0000 7f428128e640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c08d600 0x7f427c06eb20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f42780097a0 tx=0x7f427802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1fc6a96dd8eb7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.353+0000 7f4273fff640 1 -- 172.21.15.146:0/1677087103 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278037070 con 0x7f427c08d600 2024-04-25T15:23:22.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.353+0000 7f4273fff640 1 -- 172.21.15.146:0/1677087103 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427802ae50 con 0x7f427c08d600 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.353+0000 7f4273fff640 1 -- 172.21.15.146:0/1677087103 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278032660 con 0x7f427c08d600 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 -- 172.21.15.146:0/1677087103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c08d600 msgr2=0x7f427c06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c08d600 0x7f427c06eb20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f42780097a0 tx=0x7f427802a100 comp rx=0 tx=0).stop 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 -- 172.21.15.146:0/1677087103 shutdown_connections 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c0d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 --2- 172.21.15.146:0/1677087103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c08d600 0x7f427c06eb20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 -- 172.21.15.146:0/1677087103 >> 172.21.15.146:0/1677087103 conn(0x7f427c0b52f0 msgr2=0x7f427c0b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 -- 172.21.15.146:0/1677087103 shutdown_connections 2024-04-25T15:23:22.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.354+0000 7f42834f2640 1 -- 172.21.15.146:0/1677087103 wait complete. 2024-04-25T15:23:22.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.356+0000 7f42834f2640 1 Processor -- start 2024-04-25T15:23:22.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.356+0000 7f42834f2640 1 -- start start 2024-04-25T15:23:22.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f42834f2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c06f060 0x7f427c162b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f42834f2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f427c08d600 0x7f427c1e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f42834f2640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c1ea200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f42834f2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f427c0b8f00 con 0x7f427c06f060 2024-04-25T15:23:22.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f42834f2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f427c0b8d80 con 0x7f427c08d600 2024-04-25T15:23:22.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f42834f2640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f427c08da10 con 0x7f427c160d70 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c06f060 0x7f427c162b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c06f060 0x7f427c162b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56974/0 (socket says 172.21.15.146:56974) 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 -- 172.21.15.146:0/119925276 learned_addr learned my addr 172.21.15.146:0/119925276 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f4281a8f640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c1ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f4280a8d640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f427c08d600 0x7f427c1e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 -- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 msgr2=0x7f427c1ea200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f427c160d70 0x7f427c1ea200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 -- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f427c08d600 msgr2=0x7f427c1e9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.357+0000 7f428128e640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f427c08d600 0x7f427c1e9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f428128e640 1 -- 172.21.15.146:0/119925276 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c1638e0 con 0x7f427c06f060 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f4280a8d640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f427c08d600 0x7f427c1e9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f428128e640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c06f060 0x7f427c162b70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f4278035040 tx=0x7f42780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278037040 con 0x7f427c06f060 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278004030 con 0x7f427c06f060 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c163bc0 con 0x7f427c06f060 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278004400 con 0x7f427c06f060 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.358+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c164180 con 0x7f427c06f060 2024-04-25T15:23:22.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.359+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f427c06a0e0 con 0x7f427c06f060 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4278003610 con 0x7f427c06f060 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4271ffb640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42600745d0 0x7f4260076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4280a8d640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42600745d0 0x7f4260076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f42780b60d0 con 0x7f427c06f060 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4271ffb640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f426007ab60 0x7f426007cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 --> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42780093e0 con 0x7f426007ab60 2024-04-25T15:23:22.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.360+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f42780bd050 con 0x7f427c06f060 2024-04-25T15:23:22.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.361+0000 7f4281a8f640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f426007ab60 0x7f426007cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.361+0000 7f4280a8d640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42600745d0 0x7f4260076ae0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f427c1646d0 tx=0x7f426c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.361+0000 7f4281a8f640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f426007ab60 0x7f426007cff0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.362+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== osd.0 v2:172.21.15.146:6816/1105656680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42780093e0 con 0x7f426007ab60 2024-04-25T15:23:22.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 --> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f427c0b5fc0 con 0x7f426007ab60 2024-04-25T15:23:22.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f4271ffb640 1 -- 172.21.15.146:0/119925276 <== osd.0 v2:172.21.15.146:6816/1105656680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f427c0b5fc0 con 0x7f426007ab60 2024-04-25T15:23:22.376 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:22.376 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:22.376 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:22.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f426007ab60 msgr2=0x7f426007cff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6816/1105656680,v1:172.21.15.146:6817/1105656680] conn(0x7f426007ab60 0x7f426007cff0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42600745d0 msgr2=0x7f4260076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f42600745d0 0x7f4260076ae0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f427c1646d0 tx=0x7f426c009040 comp rx=0 tx=0).stop 2024-04-25T15:23:22.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c06f060 msgr2=0x7f427c162b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 --2- 172.21.15.146:0/119925276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c06f060 0x7f427c162b70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f4278035040 tx=0x7f42780046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f4281a8f640 1 -- 172.21.15.146:0/119925276 reap_dead start 2024-04-25T15:23:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 shutdown_connections 2024-04-25T15:23:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 >> 172.21.15.146:0/119925276 conn(0x7f427c0b52f0 msgr2=0x7f427c0ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 shutdown_connections 2024-04-25T15:23:22.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.375+0000 7f42834f2640 1 -- 172.21.15.146:0/119925276 wait complete. 2024-04-25T15:23:22.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T15:23:22.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.495+0000 7fc534f57640 1 Processor -- start 2024-04-25T15:23:22.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.495+0000 7fc534f57640 1 -- start start 2024-04-25T15:23:22.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc534f57640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5300d4f60 0x7fc5300d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc534f57640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc5300d0020 0x7fc5300d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc534f57640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc530163d80 0x7fc530164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc534f57640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc53006c550 con 0x7fc5300d4f60 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc534f57640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fc53006c830 con 0x7fc5300d0020 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc534f57640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc5300b9200 con 0x7fc530163d80 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc52e59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5300d4f60 0x7fc5300d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc52e59c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5300d4f60 0x7fc5300d5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56982/0 (socket says 172.21.15.146:56982) 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc52e59c640 1 -- 172.21.15.146:0/697021774 learned_addr learned my addr 172.21.15.146:0/697021774 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:22.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc52ed9d640 1 --2- 172.21.15.146:0/697021774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc530163d80 0x7fc530164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.496+0000 7fc52dd9b640 1 --2- 172.21.15.146:0/697021774 >> v2:172.21.15.177:3300/0 conn(0x7fc5300d0020 0x7fc5300d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52e59c640 1 -- 172.21.15.146:0/697021774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc530163d80 msgr2=0x7fc530164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:22.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52e59c640 1 --2- 172.21.15.146:0/697021774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc530163d80 0x7fc530164190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52e59c640 1 -- 172.21.15.146:0/697021774 >> v2:172.21.15.177:3300/0 conn(0x7fc5300d0020 msgr2=0x7fc5300d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:22.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52e59c640 1 --2- 172.21.15.146:0/697021774 >> v2:172.21.15.177:3300/0 conn(0x7fc5300d0020 0x7fc5300d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52e59c640 1 -- 172.21.15.146:0/697021774 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5300d0970 con 0x7fc5300d4f60 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52e59c640 1 --2- 172.21.15.146:0/697021774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5300d4f60 0x7fc5300d5370 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc5300b8c90 tx=0x7fc52002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7910ce08984bac62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.497+0000 7fc52d59a640 1 -- 172.21.15.146:0/697021774 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520036070 con 0x7fc5300d4f60 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc52d59a640 1 -- 172.21.15.146:0/697021774 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc520031600 con 0x7fc5300d4f60 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc52d59a640 1 -- 172.21.15.146:0/697021774 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200319d0 con 0x7fc5300d4f60 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 -- 172.21.15.146:0/697021774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5300d4f60 msgr2=0x7fc5300d5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 --2- 172.21.15.146:0/697021774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5300d4f60 0x7fc5300d5370 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc5300b8c90 tx=0x7fc52002a570 comp rx=0 tx=0).stop 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 -- 172.21.15.146:0/697021774 shutdown_connections 2024-04-25T15:23:22.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 --2- 172.21.15.146:0/697021774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc530163d80 0x7fc530164190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 --2- 172.21.15.146:0/697021774 >> v2:172.21.15.177:3300/0 conn(0x7fc5300d0020 0x7fc5300d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 --2- 172.21.15.146:0/697021774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5300d4f60 0x7fc5300d5370 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 -- 172.21.15.146:0/697021774 >> 172.21.15.146:0/697021774 conn(0x7fc5300d2450 msgr2=0x7fc5300d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 -- 172.21.15.146:0/697021774 shutdown_connections 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.498+0000 7fc534f57640 1 -- 172.21.15.146:0/697021774 wait complete. 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.500+0000 7fc534f57640 1 Processor -- start 2024-04-25T15:23:22.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.500+0000 7fc534f57640 1 -- start start 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.501+0000 7fc534f57640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5300d0020 0x7fc53015af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.501+0000 7fc534f57640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5300d4f60 0x7fc5300cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.501+0000 7fc534f57640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 0x7fc5300cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.501+0000 7fc534f57640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5300d15f0 con 0x7fc530163d80 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.501+0000 7fc534f57640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5300d1070 con 0x7fc5300d0020 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.501+0000 7fc534f57640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fc5300b5150 con 0x7fc5300d4f60 2024-04-25T15:23:22.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 0x7fc5300cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 0x7fc5300cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56988/0 (socket says 172.21.15.146:56988) 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 -- 172.21.15.146:0/3455329098 learned_addr learned my addr 172.21.15.146:0/3455329098 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52dd9b640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5300d4f60 0x7fc5300cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52e59c640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5300d0020 0x7fc53015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 -- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5300d4f60 msgr2=0x7fc5300cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5300d4f60 0x7fc5300cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 -- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5300d0020 msgr2=0x7fc53015af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5300d0020 0x7fc53015af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 -- 172.21.15.146:0/3455329098 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5300cd520 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc52ed9d640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 0x7fc5300cb610 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e7f0 tx=0x7fc52800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528017070 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc52800c730 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5300ca580 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52800cb20 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.502+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53015ce30 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.503+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53006a0e0 con 0x7fc530163d80 2024-04-25T15:23:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.504+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fc52801a070 con 0x7fc530163d80 2024-04-25T15:23:22.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.504+0000 7fc51effd640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc510074600 0x7fc510076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.504+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fc528094fb0 con 0x7fc530163d80 2024-04-25T15:23:22.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.504+0000 7fc52e59c640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc510074600 0x7fc510076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.505+0000 7fc52e59c640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc510074600 0x7fc510076b10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc520033000 tx=0x7fc520009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.509+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc52805f190 con 0x7fc530163d80 2024-04-25T15:23:22.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.685+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fc5300ca310 con 0x7fc510074600 2024-04-25T15:23:22.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.686+0000 7fc51effd640 1 -- 172.21.15.146:0/3455329098 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5300ca310 con 0x7fc510074600 2024-04-25T15:23:22.687 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T15:23:22.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc510074600 msgr2=0x7fc510076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc510074600 0x7fc510076b10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc520033000 tx=0x7fc520009000 comp rx=0 tx=0).stop 2024-04-25T15:23:22.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 msgr2=0x7fc5300cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 0x7fc5300cb610 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e7f0 tx=0x7fc52800ecb0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 shutdown_connections 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fc510074600 0x7fc510076b10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc530163d80 0x7fc5300cb610 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5300d4f60 0x7fc5300cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 --2- 172.21.15.146:0/3455329098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5300d0020 0x7fc53015af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 >> 172.21.15.146:0/3455329098 conn(0x7fc5300d2450 msgr2=0x7fc5300bac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 shutdown_connections 2024-04-25T15:23:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.688+0000 7fc534f57640 1 -- 172.21.15.146:0/3455329098 wait complete. 2024-04-25T15:23:22.700 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T15:23:22.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:22.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.812+0000 7f9f5f1f3640 1 Processor -- start 2024-04-25T15:23:22.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.812+0000 7f9f5f1f3640 1 -- start start 2024-04-25T15:23:22.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.813+0000 7f9f5f1f3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f580cea60 0x7f9f580cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.813+0000 7f9f5f1f3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f580b5f60 con 0x7f9f5806de40 2024-04-25T15:23:22.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.813+0000 7f9f5f1f3640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f5809ce00 con 0x7f9f580cf3b0 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.813+0000 7f9f5f1f3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9f580b6620 con 0x7f9f580cea60 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f4ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f580cea60 0x7f9f580cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f5cf8f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9f5806de40 0x7f9f5806eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43534/0 (socket says 172.21.15.146:43534) 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f5cf8f640 1 -- 172.21.15.146:0/160803203 learned_addr learned my addr 172.21.15.146:0/160803203 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f4ffff640 1 -- 172.21.15.146:0/160803203 >> v1:172.21.15.177:6789/0 conn(0x7f9f580cf3b0 legacy=0x7f9f580d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f4ffff640 1 -- 172.21.15.146:0/160803203 >> v1:172.21.15.146:6789/0 conn(0x7f9f5806de40 legacy=0x7f9f5806eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f4ffff640 1 -- 172.21.15.146:0/160803203 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5809d0e0 con 0x7f9f580cea60 2024-04-25T15:23:22.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.814+0000 7f9f4f7fe640 1 -- 172.21.15.146:0/160803203 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823973708 0 0) 0x7f9f580b5f60 con 0x7f9f5806de40 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f4ffff640 1 --2- 172.21.15.146:0/160803203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f580cea60 0x7f9f580cee70 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f9f480097a0 tx=0x7f9f4802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181afbbf46a01e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f4f7fe640 1 -- 172.21.15.146:0/160803203 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48037070 con 0x7f9f580cea60 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f4f7fe640 1 -- 172.21.15.146:0/160803203 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f48032440 con 0x7f9f580cea60 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f4f7fe640 1 -- 172.21.15.146:0/160803203 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48032810 con 0x7f9f580cea60 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/160803203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f580cea60 msgr2=0x7f9f580cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f5f1f3640 1 --2- 172.21.15.146:0/160803203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f580cea60 0x7f9f580cee70 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f9f480097a0 tx=0x7f9f4802a230 comp rx=0 tx=0).stop 2024-04-25T15:23:22.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.815+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/160803203 shutdown_connections 2024-04-25T15:23:22.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.816+0000 7f9f5f1f3640 1 --2- 172.21.15.146:0/160803203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f580cea60 0x7f9f580cee70 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.816+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/160803203 >> 172.21.15.146:0/160803203 conn(0x7f9f580b52c0 msgr2=0x7f9f580b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.816+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/160803203 shutdown_connections 2024-04-25T15:23:22.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.816+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/160803203 wait complete. 2024-04-25T15:23:22.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.817+0000 7f9f5f1f3640 1 Processor -- start 2024-04-25T15:23:22.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.817+0000 7f9f5f1f3640 1 -- start start 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5f1f3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5806de40 0x7f9f580b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5f1f3640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f580cea60 0x7f9f581fa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5f1f3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580cf3b0 0x7f9f5809dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5f1f3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5806e3d0 con 0x7f9f5806de40 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5f1f3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5806e250 con 0x7f9f580cf3b0 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5f1f3640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f9f580cf7c0 con 0x7f9f580cea60 2024-04-25T15:23:22.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5806de40 0x7f9f580b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5806de40 0x7f9f580b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57004/0 (socket says 172.21.15.146:57004) 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 -- 172.21.15.146:0/552339271 learned_addr learned my addr 172.21.15.146:0/552339271 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f4ffff640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f580cea60 0x7f9f581fa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5d790640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580cf3b0 0x7f9f5809dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 -- 172.21.15.146:0/552339271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f580cea60 msgr2=0x7f9f581fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f580cea60 0x7f9f581fa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 -- 172.21.15.146:0/552339271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580cf3b0 msgr2=0x7f9f5809dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580cf3b0 0x7f9f5809dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 -- 172.21.15.146:0/552339271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f580a06a0 con 0x7f9f5806de40 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.819+0000 7f9f5cf8f640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5806de40 0x7f9f580b3f10 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400c3c0 tx=0x7f9f5400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.820+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54017070 con 0x7f9f5806de40 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.820+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f54007500 con 0x7f9f5806de40 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.820+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f580a0980 con 0x7f9f5806de40 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.820+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540059e0 con 0x7f9f5806de40 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.820+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f580a0f40 con 0x7f9f5806de40 2024-04-25T15:23:22.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.820+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f580a1320 con 0x7f9f5806de40 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.821+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f9f5401a070 con 0x7f9f5806de40 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4d7fa640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f44074600 0x7f9f44076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4ffff640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f44074600 0x7f9f44076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f9f54094b80 con 0x7f9f5806de40 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4d7fa640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7f9f4407ab90 0x7f9f4407d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 --> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f4407d6f0 con 0x7f9f4407ab90 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f9f54095010 con 0x7f9f5806de40 2024-04-25T15:23:22.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f5d790640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7f9f4407ab90 0x7f9f4407d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f4ffff640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f44074600 0x7f9f44076b10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9f48034000 tx=0x7f9f4802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.822+0000 7f9f5d790640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7f9f4407ab90 0x7f9f4407d020 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.823+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== osd.1 v2:172.21.15.146:6824/3545155054 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f4407d6f0 con 0x7f9f4407ab90 2024-04-25T15:23:22.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.836+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 --> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9f580a0120 con 0x7f9f4407ab90 2024-04-25T15:23:22.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.836+0000 7f9f4d7fa640 1 -- 172.21.15.146:0/552339271 <== osd.1 v2:172.21.15.146:6824/3545155054 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9f580a0120 con 0x7f9f4407ab90 2024-04-25T15:23:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:22.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7f9f4407ab90 msgr2=0x7f9f4407d020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6824/3545155054,v1:172.21.15.146:6825/3545155054] conn(0x7f9f4407ab90 0x7f9f4407d020 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f44074600 msgr2=0x7f9f44076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f9f44074600 0x7f9f44076b10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9f48034000 tx=0x7f9f4802e000 comp rx=0 tx=0).stop 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5806de40 msgr2=0x7f9f580b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 --2- 172.21.15.146:0/552339271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5806de40 0x7f9f580b3f10 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400c3c0 tx=0x7f9f5400c880 comp rx=0 tx=0).stop 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5d790640 1 -- 172.21.15.146:0/552339271 reap_dead start 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 shutdown_connections 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 >> 172.21.15.146:0/552339271 conn(0x7f9f580b52c0 msgr2=0x7f9f580b67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 shutdown_connections 2024-04-25T15:23:22.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.837+0000 7f9f5f1f3640 1 -- 172.21.15.146:0/552339271 wait complete. 2024-04-25T15:23:22.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T15:23:22.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.958+0000 7f16c5ee7640 1 Processor -- start 2024-04-25T15:23:22.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.958+0000 7f16c5ee7640 1 -- start start 2024-04-25T15:23:22.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16c5ee7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c006f4d0 0x7f16c006fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16c5ee7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16c0160020 0x7f16c00d1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16c5ee7640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16c00d1cb0 0x7f16c00b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16c5ee7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f16c00d0e00 con 0x7f16c0160020 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16c5ee7640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f16c00b2b50 con 0x7f16c00d1cb0 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16c5ee7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f16c00b2e30 con 0x7f16c006f4d0 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16beffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16c0160020 0x7f16c00d1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16beffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16c0160020 0x7f16c00d1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57006/0 (socket says 172.21.15.146:57006) 2024-04-25T15:23:22.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16beffd640 1 -- 172.21.15.146:0/2536971728 learned_addr learned my addr 172.21.15.146:0/2536971728 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16bf7fe640 1 --2- 172.21.15.146:0/2536971728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c006f4d0 0x7f16c006fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.959+0000 7f16bffff640 1 --2- 172.21.15.146:0/2536971728 >> v2:172.21.15.177:3300/0 conn(0x7f16c00d1cb0 0x7f16c00b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.960+0000 7f16beffd640 1 -- 172.21.15.146:0/2536971728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c006f4d0 msgr2=0x7f16c006fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.960+0000 7f16beffd640 1 --2- 172.21.15.146:0/2536971728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c006f4d0 0x7f16c006fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.960+0000 7f16beffd640 1 -- 172.21.15.146:0/2536971728 >> v2:172.21.15.177:3300/0 conn(0x7f16c00d1cb0 msgr2=0x7f16c00b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.960+0000 7f16beffd640 1 --2- 172.21.15.146:0/2536971728 >> v2:172.21.15.177:3300/0 conn(0x7f16c00d1cb0 0x7f16c00b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.960+0000 7f16beffd640 1 -- 172.21.15.146:0/2536971728 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c006f260 con 0x7f16c0160020 2024-04-25T15:23:22.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.961+0000 7f16beffd640 1 --2- 172.21.15.146:0/2536971728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c00d1770 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f16c006db50 tx=0x7f16b402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bfaaa3f1baa003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.961+0000 7f16be7fc640 1 -- 172.21.15.146:0/2536971728 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4037070 con 0x7f16c0160020 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.961+0000 7f16be7fc640 1 -- 172.21.15.146:0/2536971728 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b4032440 con 0x7f16c0160020 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.961+0000 7f16be7fc640 1 -- 172.21.15.146:0/2536971728 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4032810 con 0x7f16c0160020 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.961+0000 7f16c5ee7640 1 -- 172.21.15.146:0/2536971728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 msgr2=0x7f16c00d1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.961+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/2536971728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c00d1770 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f16c006db50 tx=0x7f16b402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 -- 172.21.15.146:0/2536971728 shutdown_connections 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/2536971728 >> v2:172.21.15.177:3300/0 conn(0x7f16c00d1cb0 0x7f16c00b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/2536971728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c00d1770 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/2536971728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c006f4d0 0x7f16c006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 -- 172.21.15.146:0/2536971728 >> 172.21.15.146:0/2536971728 conn(0x7f16c00d0240 msgr2=0x7f16c00d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:22.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 -- 172.21.15.146:0/2536971728 shutdown_connections 2024-04-25T15:23:22.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.962+0000 7f16c5ee7640 1 -- 172.21.15.146:0/2536971728 wait complete. 2024-04-25T15:23:22.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.963+0000 7f16c5ee7640 1 Processor -- start 2024-04-25T15:23:22.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.964+0000 7f16c5ee7640 1 -- start start 2024-04-25T15:23:22.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16c5ee7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16c006f4d0 0x7f16c00ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16c5ee7640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c00d1cb0 0x7f16c015edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16c5ee7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c01f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16c5ee7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f16c01605b0 con 0x7f16c0160020 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16c5ee7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f16c0160430 con 0x7f16c006f4d0 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16c5ee7640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f16c00b5780 con 0x7f16c00d1cb0 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c01f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c01f9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57018/0 (socket says 172.21.15.146:57018) 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 -- 172.21.15.146:0/4059361763 learned_addr learned my addr 172.21.15.146:0/4059361763 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16beffd640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c00d1cb0 0x7f16c015edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bf7fe640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16c006f4d0 0x7f16c00ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 -- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c00d1cb0 msgr2=0x7f16c015edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c00d1cb0 0x7f16c015edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 -- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16c006f4d0 msgr2=0x7f16c00ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16c006f4d0 0x7f16c00ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 -- 172.21.15.146:0/4059361763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c00cba40 con 0x7f16c0160020 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.965+0000 7f16bffff640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c01f9d20 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f16b800ef90 tx=0x7f16b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.966+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8018020 con 0x7f16c0160020 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.966+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b8014480 con 0x7f16c0160020 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.966+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c00caec0 con 0x7f16c0160020 2024-04-25T15:23:22.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.966+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8009e60 con 0x7f16c0160020 2024-04-25T15:23:22.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.966+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c00cb480 con 0x7f16c0160020 2024-04-25T15:23:22.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.967+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c0068f10 con 0x7f16c0160020 2024-04-25T15:23:22.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.967+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f16b8005d10 con 0x7f16c0160020 2024-04-25T15:23:22.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.968+0000 7f16a7fff640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16ac0745d0 0x7f16ac076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:22.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.968+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f16b8095e10 con 0x7f16c0160020 2024-04-25T15:23:22.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.968+0000 7f16bf7fe640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16ac0745d0 0x7f16ac076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:22.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.968+0000 7f16bf7fe640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16ac0745d0 0x7f16ac076ae0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009aa0 tx=0x7f16b0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:22.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:22.973+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16c0068f10 con 0x7f16c0160020 2024-04-25T15:23:23.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.151+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f16c00cac50 con 0x7f16ac0745d0 2024-04-25T15:23:23.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.152+0000 7f16a7fff640 1 -- 172.21.15.146:0/4059361763 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f16c00cac50 con 0x7f16ac0745d0 2024-04-25T15:23:23.153 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T15:23:23.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16ac0745d0 msgr2=0x7f16ac076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16ac0745d0 0x7f16ac076ae0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009aa0 tx=0x7f16b0009190 comp rx=0 tx=0).stop 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 msgr2=0x7f16c01f9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c01f9d20 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f16b800ef90 tx=0x7f16b800c490 comp rx=0 tx=0).stop 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 shutdown_connections 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f16ac0745d0 0x7f16ac076ae0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16c0160020 0x7f16c01f9d20 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16c00d1cb0 0x7f16c015edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 --2- 172.21.15.146:0/4059361763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16c006f4d0 0x7f16c00ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 >> 172.21.15.146:0/4059361763 conn(0x7f16c00d0240 msgr2=0x7f16c00d34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 shutdown_connections 2024-04-25T15:23:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.154+0000 7f16c5ee7640 1 -- 172.21.15.146:0/4059361763 wait complete. 2024-04-25T15:23:23.166 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T15:23:23.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:23.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.277+0000 7f209e346640 1 Processor -- start 2024-04-25T15:23:23.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.277+0000 7f209e346640 1 -- start start 2024-04-25T15:23:23.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f209e346640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 0x7f209815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f209e346640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2098162050 0x7f209806c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f209e346640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20980cc890 con 0x7f20980d0a20 2024-04-25T15:23:23.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f209e346640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f209815bde0 con 0x7f2098162050 2024-04-25T15:23:23.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f209e346640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f20980ccb70 con 0x7f20980cc2f0 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 0x7f209815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 0x7f209815b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:35432/0 (socket says 172.21.15.146:35432) 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 -- 172.21.15.146:0/3690453801 learned_addr learned my addr 172.21.15.146:0/3690453801 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f209c8e3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2098162050 0x7f209806c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 -- 172.21.15.146:0/3690453801 >> v2:172.21.15.177:3300/0 conn(0x7f2098162050 msgr2=0x7f209806c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 --2- 172.21.15.146:0/3690453801 >> v2:172.21.15.177:3300/0 conn(0x7f2098162050 0x7f209806c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 -- 172.21.15.146:0/3690453801 >> v1:172.21.15.177:6789/0 conn(0x7f20980d0a20 legacy=0x7f2098161940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.279+0000 7f2097fff640 1 -- 172.21.15.146:0/3690453801 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20980cce50 con 0x7f20980cc2f0 2024-04-25T15:23:23.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.280+0000 7f209c8e3640 1 --2- 172.21.15.146:0/3690453801 >> v2:172.21.15.177:3300/0 conn(0x7f2098162050 0x7f209806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:23.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.280+0000 7f2097fff640 1 --2- 172.21.15.146:0/3690453801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 0x7f209815b8a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2088009870 tx=0x7f208802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa69e66d73bdf22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.280+0000 7f2096ffd640 1 -- 172.21.15.146:0/3690453801 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088037070 con 0x7f20980cc2f0 2024-04-25T15:23:23.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.280+0000 7f2096ffd640 1 -- 172.21.15.146:0/3690453801 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2088032440 con 0x7f20980cc2f0 2024-04-25T15:23:23.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.280+0000 7f2096ffd640 1 -- 172.21.15.146:0/3690453801 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088032810 con 0x7f20980cc2f0 2024-04-25T15:23:23.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 -- 172.21.15.146:0/3690453801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 msgr2=0x7f209815b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 --2- 172.21.15.146:0/3690453801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 0x7f209815b8a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2088009870 tx=0x7f208802a270 comp rx=0 tx=0).stop 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 -- 172.21.15.146:0/3690453801 shutdown_connections 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 --2- 172.21.15.146:0/3690453801 >> v2:172.21.15.177:3300/0 conn(0x7f2098162050 0x7f209806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 --2- 172.21.15.146:0/3690453801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20980cc2f0 0x7f209815b8a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 -- 172.21.15.146:0/3690453801 >> 172.21.15.146:0/3690453801 conn(0x7f20980ce470 msgr2=0x7f20980ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 -- 172.21.15.146:0/3690453801 shutdown_connections 2024-04-25T15:23:23.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.281+0000 7f209e346640 1 -- 172.21.15.146:0/3690453801 wait complete. 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.283+0000 7f209e346640 1 Processor -- start 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.283+0000 7f209e346640 1 -- start start 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.284+0000 7f209e346640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20980cc2f0 0x7f209809ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.284+0000 7f209e346640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20980d0a20 0x7f209809c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.284+0000 7f209e346640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20981f4040 0x7f2098159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.284+0000 7f209e346640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20980d0fb0 con 0x7f20980cc2f0 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.284+0000 7f209e346640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20980d0e30 con 0x7f20980d0a20 2024-04-25T15:23:23.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.284+0000 7f209e346640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f209806cd70 con 0x7f20981f4040 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20980cc2f0 0x7f209809ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20980cc2f0 0x7f209809ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57026/0 (socket says 172.21.15.146:57026) 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 -- 172.21.15.146:0/3853523723 learned_addr learned my addr 172.21.15.146:0/3853523723 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f209c8e3640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20981f4040 0x7f2098159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f20977fe640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20980d0a20 0x7f209809c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 -- 172.21.15.146:0/3853523723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20981f4040 msgr2=0x7f2098159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20981f4040 0x7f2098159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 -- 172.21.15.146:0/3853523723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20980d0a20 msgr2=0x7f209809c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20980d0a20 0x7f209809c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 -- 172.21.15.146:0/3853523723 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209815a140 con 0x7f20980cc2f0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2097fff640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20980cc2f0 0x7f209809ea50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2088000c00 tx=0x7f20880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088037040 con 0x7f20980cc2f0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2088042070 con 0x7f20980cc2f0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209815d940 con 0x7f20980cc2f0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20880308a0 con 0x7f20980cc2f0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.285+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f209815df00 con 0x7f20980cc2f0 2024-04-25T15:23:23.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.286+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f209809cb10 con 0x7f20980cc2f0 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.287+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f2088030a40 con 0x7f20980cc2f0 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.287+0000 7f2094ff9640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20840745d0 0x7f2084076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f20880b6580 con 0x7f20980cc2f0 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f20977fe640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20840745d0 0x7f2084076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f2094ff9640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7f208407ab60 0x7f208407cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 --> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2088034e90 con 0x7f208407ab60 2024-04-25T15:23:23.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f20880b6a10 con 0x7f20980cc2f0 2024-04-25T15:23:23.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f209c8e3640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7f208407ab60 0x7f208407cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f20977fe640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20840745d0 0x7f2084076ae0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f209806cb10 tx=0x7f208c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.288+0000 7f209c8e3640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7f208407ab60 0x7f208407cff0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.289+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== osd.2 v2:172.21.15.146:6800/3921051373 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2088034e90 con 0x7f208407ab60 2024-04-25T15:23:23.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.302+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 --> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2098071d60 con 0x7f208407ab60 2024-04-25T15:23:23.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.302+0000 7f2094ff9640 1 -- 172.21.15.146:0/3853523723 <== osd.2 v2:172.21.15.146:6800/3921051373 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2098071d60 con 0x7f208407ab60 2024-04-25T15:23:23.303 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:23.303 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:23.303 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:23.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.302+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7f208407ab60 msgr2=0x7f208407cff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.302+0000 7f209e346640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6800/3921051373,v1:172.21.15.146:6801/3921051373] conn(0x7f208407ab60 0x7f208407cff0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.302+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20840745d0 msgr2=0x7f2084076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f20840745d0 0x7f2084076ae0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f209806cb10 tx=0x7f208c009040 comp rx=0 tx=0).stop 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20980cc2f0 msgr2=0x7f209809ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 --2- 172.21.15.146:0/3853523723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20980cc2f0 0x7f209809ea50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2088000c00 tx=0x7f20880046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209c8e3640 1 -- 172.21.15.146:0/3853523723 reap_dead start 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 shutdown_connections 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 >> 172.21.15.146:0/3853523723 conn(0x7f20980ce470 msgr2=0x7f20980b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 shutdown_connections 2024-04-25T15:23:23.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.303+0000 7f209e346640 1 -- 172.21.15.146:0/3853523723 wait complete. 2024-04-25T15:23:23.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T15:23:23.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.421+0000 7f38d20e0640 1 Processor -- start 2024-04-25T15:23:23.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.421+0000 7f38d20e0640 1 -- start start 2024-04-25T15:23:23.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38d20e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0b4b10 0x7f38cc0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38d20e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38cc0d2e20 0x7f38cc0af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38d20e0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38cc0cfea0 con 0x7f38cc0cfa90 2024-04-25T15:23:23.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38d20e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f38cc0b4f20 con 0x7f38cc0d2e20 2024-04-25T15:23:23.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38d20e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc0b3a70 con 0x7f38cc0b4b10 2024-04-25T15:23:23.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38cc0d2e20 0x7f38cc0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38cc0d2e20 0x7f38cc0af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57042/0 (socket says 172.21.15.146:57042) 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 -- 172.21.15.146:0/3621807728 learned_addr learned my addr 172.21.15.146:0/3621807728 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cb7fe640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0b4b10 0x7f38cc0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 -- 172.21.15.146:0/3621807728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0b4b10 msgr2=0x7f38cc0cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0b4b10 0x7f38cc0cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 -- 172.21.15.146:0/3621807728 >> v1:172.21.15.177:6789/0 conn(0x7f38cc0cfa90 legacy=0x7f38cc0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cbfff640 1 -- 172.21.15.146:0/3621807728 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc0b3d50 con 0x7f38cc0d2e20 2024-04-25T15:23:23.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.423+0000 7f38cb7fe640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0b4b10 0x7f38cc0cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:23.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.424+0000 7f38cbfff640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0d2e20 0x7f38cc0af490 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009e50 tx=0x7f38c402a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe14b9478d09a84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.424+0000 7f38ca7fc640 1 -- 172.21.15.146:0/3621807728 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4036070 con 0x7f38cc0d2e20 2024-04-25T15:23:23.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.424+0000 7f38ca7fc640 1 -- 172.21.15.146:0/3621807728 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c4031600 con 0x7f38cc0d2e20 2024-04-25T15:23:23.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.424+0000 7f38ca7fc640 1 -- 172.21.15.146:0/3621807728 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c40319d0 con 0x7f38cc0d2e20 2024-04-25T15:23:23.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 -- 172.21.15.146:0/3621807728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0d2e20 msgr2=0x7f38cc0af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0d2e20 0x7f38cc0af490 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009e50 tx=0x7f38c402a630 comp rx=0 tx=0).stop 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 -- 172.21.15.146:0/3621807728 shutdown_connections 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0d2e20 0x7f38cc0af490 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 --2- 172.21.15.146:0/3621807728 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0b4b10 0x7f38cc0cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 -- 172.21.15.146:0/3621807728 >> 172.21.15.146:0/3621807728 conn(0x7f38cc0b73b0 msgr2=0x7f38cc0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 -- 172.21.15.146:0/3621807728 shutdown_connections 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.425+0000 7f38d20e0640 1 -- 172.21.15.146:0/3621807728 wait complete. 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.426+0000 7f38d20e0640 1 Processor -- start 2024-04-25T15:23:23.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.427+0000 7f38d20e0640 1 -- start start 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38d20e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 0x7f38cc0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38d20e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38cc0cfa90 0x7f38cc0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38d20e0640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0d2e20 0x7f38cc0b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38d20e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc0b45f0 con 0x7f38cc0b4b10 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38d20e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc0b4310 con 0x7f38cc0cfa90 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38d20e0640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc0cefc0 con 0x7f38cc0d2e20 2024-04-25T15:23:23.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 0x7f38cc0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 0x7f38cc0c8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57058/0 (socket says 172.21.15.146:57058) 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 -- 172.21.15.146:0/4101310302 learned_addr learned my addr 172.21.15.146:0/4101310302 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cbfff640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0d2e20 0x7f38cc0b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 -- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0d2e20 msgr2=0x7f38cc0b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0d2e20 0x7f38cc0b0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 -- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38cc0cfa90 msgr2=0x7f38cc0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38cc0cfa90 0x7f38cc0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.428+0000 7f38cb7fe640 1 -- 172.21.15.146:0/4101310302 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc15e7a0 con 0x7f38cc0b4b10 2024-04-25T15:23:23.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.429+0000 7f38cb7fe640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 0x7f38cc0c8b80 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00c7f0 tx=0x7f38bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.429+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc005b90 con 0x7f38cc0b4b10 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.429+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38bc005d30 con 0x7f38cc0b4b10 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.429+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc15ea80 con 0x7f38cc0b4b10 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.429+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc006270 con 0x7f38cc0b4b10 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.429+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc0d0ef0 con 0x7f38cc0b4b10 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.430+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38cc06a0e0 con 0x7f38cc0b4b10 2024-04-25T15:23:23.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.430+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f38bc006620 con 0x7f38cc0b4b10 2024-04-25T15:23:23.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.431+0000 7f38b3fff640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f38b80745d0 0x7f38b8076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.431+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f38bc095490 con 0x7f38cc0b4b10 2024-04-25T15:23:23.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.431+0000 7f38caffd640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f38b80745d0 0x7f38b8076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.431+0000 7f38caffd640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f38b80745d0 0x7f38b8076ae0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0c5900 tx=0x7f38c0006be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.435+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38bc015020 con 0x7f38cc0b4b10 2024-04-25T15:23:23.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.609+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f38cc15e550 con 0x7f38b80745d0 2024-04-25T15:23:23.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.609+0000 7f38b3fff640 1 -- 172.21.15.146:0/4101310302 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f38cc15e550 con 0x7f38b80745d0 2024-04-25T15:23:23.610 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T15:23:23.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f38b80745d0 msgr2=0x7f38b8076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f38b80745d0 0x7f38b8076ae0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0c5900 tx=0x7f38c0006be0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 msgr2=0x7f38cc0c8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 0x7f38cc0c8b80 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00c7f0 tx=0x7f38bc00ccb0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 shutdown_connections 2024-04-25T15:23:23.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f38b80745d0 0x7f38b8076ae0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38cc0d2e20 0x7f38cc0b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38cc0cfa90 0x7f38cc0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 --2- 172.21.15.146:0/4101310302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38cc0b4b10 0x7f38cc0c8b80 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 >> 172.21.15.146:0/4101310302 conn(0x7f38cc0b73b0 msgr2=0x7f38cc0b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 shutdown_connections 2024-04-25T15:23:23.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.611+0000 7f38d20e0640 1 -- 172.21.15.146:0/4101310302 wait complete. 2024-04-25T15:23:23.623 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T15:23:23.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:23.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.735+0000 7f5c10ba8640 1 Processor -- start 2024-04-25T15:23:23.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.735+0000 7f5c10ba8640 1 -- start start 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c10ba8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5c0c163c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c10ba8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0cea30 con 0x7f5c0c160000 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c10ba8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c164290 con 0x7f5c0c0d1400 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c10ba8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c164570 con 0x7f5c0c0ce620 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c09d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5c0c160000 0x7f5c0c0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43546/0 (socket says 172.21.15.146:43546) 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c09d9b640 1 -- 172.21.15.146:0/3901341202 learned_addr learned my addr 172.21.15.146:0/3901341202 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c0a59c640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5c0c163c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606105587 0 0) 0x7f5c0c0cea30 con 0x7f5c0c160000 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.737+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf4003520 con 0x7f5c0c160000 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1681847886 0 0) 0x7f5c0c164290 con 0x7f5c0c0d1400 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c0c0cea30 con 0x7f5c0c0d1400 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2560663114 0 0) 0x7f5bf4003520 con 0x7f5c0c160000 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 msgr2=0x7f5c0c163c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5c0c163c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 >> v1:172.21.15.177:6789/0 conn(0x7f5c0c0d1400 legacy=0x7f5c0c0d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c164850 con 0x7f5c0c160000 2024-04-25T15:23:23.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7f5c00001670 con 0x7f5c0c160000 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 >> v1:172.21.15.146:6789/0 conn(0x7f5c0c160000 legacy=0x7f5c0c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4003b70 0x7f5bf4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf4004800 0x7f5bf4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5bf4005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c164570 con 0x7f5bf4004800 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf400bb80 con 0x7f5bf4003b70 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf400bd80 con 0x7f5c0c0ce620 2024-04-25T15:23:23.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf4004800 0x7f5bf4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0a59c640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4003b70 0x7f5bf4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0ad9d640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5bf4005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c09d9b640 1 -- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 msgr2=0x7f5bf4005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5bf4005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c09d9b640 1 -- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4003b70 msgr2=0x7f5bf4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4003b70 0x7f5bf4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c09d9b640 1 -- 172.21.15.146:0/3901341202 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf4003520 con 0x7f5bf4004800 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.738+0000 7f5c0ad9d640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c0ce620 0x7f5bf4005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:23.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.739+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf4004800 0x7f5bf4004c10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c074720 tx=0x7f5bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9abbc9bcd2bc8dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.739+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007ce0 con 0x7f5bf4004800 2024-04-25T15:23:23.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.739+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c00005ad0 con 0x7f5bf4004800 2024-04-25T15:23:23.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.739+0000 7f5c0959a640 1 -- 172.21.15.146:0/3901341202 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00007070 con 0x7f5bf4004800 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c10ba8640 1 -- 172.21.15.146:0/3901341202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf4004800 msgr2=0x7f5bf4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c10ba8640 1 --2- 172.21.15.146:0/3901341202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf4004800 0x7f5bf4004c10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c074720 tx=0x7f5bf4005830 comp rx=0 tx=0).stop 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c0ad9d640 1 -- 172.21.15.146:0/3901341202 reap_dead start 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c10ba8640 1 -- 172.21.15.146:0/3901341202 shutdown_connections 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c10ba8640 1 -- 172.21.15.146:0/3901341202 >> 172.21.15.146:0/3901341202 conn(0x7f5c0c0d02b0 msgr2=0x7f5c0c0d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c10ba8640 1 -- 172.21.15.146:0/3901341202 shutdown_connections 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.740+0000 7f5c10ba8640 1 -- 172.21.15.146:0/3901341202 wait complete. 2024-04-25T15:23:23.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.742+0000 7f5c10ba8640 1 Processor -- start 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.742+0000 7f5c10ba8640 1 -- start start 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c10ba8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4004800 0x7f5c0c0cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c10ba8640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c15cfd0 0x7f5c0c15d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c10ba8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0c0b1290 0x7f5c0c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c10ba8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c163c00 con 0x7f5c0c0b1290 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c10ba8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c160410 con 0x7f5bf4004800 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c10ba8640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c163d80 con 0x7f5c0c15cfd0 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0c0b1290 0x7f5c0c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0c0b1290 0x7f5c0c0b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57066/0 (socket says 172.21.15.146:57066) 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 -- 172.21.15.146:0/1141112740 learned_addr learned my addr 172.21.15.146:0/1141112740 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c15cfd0 0x7f5c0c15d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0a59c640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4004800 0x7f5c0c0cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 -- 172.21.15.146:0/1141112740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c15cfd0 msgr2=0x7f5c0c15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c0c15cfd0 0x7f5c0c15d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 -- 172.21.15.146:0/1141112740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4004800 msgr2=0x7f5c0c0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bf4004800 0x7f5c0c0cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.743+0000 7f5c0ad9d640 1 -- 172.21.15.146:0/1141112740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c0b1be0 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.744+0000 7f5c0ad9d640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0c0b1290 0x7f5c0c0b16a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5c04007ae0 tx=0x7f5c0400e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.744+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04019070 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.744+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c040052e0 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.744+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04003830 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.744+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c1f7830 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.744+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c15e000 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.745+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c0c06a0e0 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.745+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f5c0401c070 con 0x7f5c0c0b1290 2024-04-25T15:23:23.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5bfaffd640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bec074600 0x7f5bec076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f5c04094eb0 con 0x7f5c0c0b1290 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5c0a59c640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bec074600 0x7f5bec076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5bfaffd640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5bec07ab90 0x7f5bec07d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 --> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bec07d6f0 con 0x7f5bec07ab90 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f5c04095340 con 0x7f5c0c0b1290 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5bec07ab90 0x7f5bec07d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.746+0000 7f5c0a59c640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bec074600 0x7f5bec076b10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c074a00 tx=0x7f5bfc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.747+0000 7f5c09d9b640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5bec07ab90 0x7f5bec07d020 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.747+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== osd.3 v2:172.21.15.146:6808/2287149109 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5bec07d6f0 con 0x7f5bec07ab90 2024-04-25T15:23:23.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.760+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 --> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5c0c15dad0 con 0x7f5bec07ab90 2024-04-25T15:23:23.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.760+0000 7f5bfaffd640 1 -- 172.21.15.146:0/1141112740 <== osd.3 v2:172.21.15.146:6808/2287149109 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5c0c15dad0 con 0x7f5bec07ab90 2024-04-25T15:23:23.761 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:23.761 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:23.762 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:23.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5bec07ab90 msgr2=0x7f5bec07d020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6808/2287149109,v1:172.21.15.146:6809/2287149109] conn(0x7f5bec07ab90 0x7f5bec07d020 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bec074600 msgr2=0x7f5bec076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f5bec074600 0x7f5bec076b10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c074a00 tx=0x7f5bfc00b040 comp rx=0 tx=0).stop 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0c0b1290 msgr2=0x7f5c0c0b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 --2- 172.21.15.146:0/1141112740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c0c0b1290 0x7f5c0c0b16a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5c04007ae0 tx=0x7f5c0400e890 comp rx=0 tx=0).stop 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c0ad9d640 1 -- 172.21.15.146:0/1141112740 reap_dead start 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 shutdown_connections 2024-04-25T15:23:23.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 >> 172.21.15.146:0/1141112740 conn(0x7f5c0c0d02b0 msgr2=0x7f5c0c1625a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 shutdown_connections 2024-04-25T15:23:23.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.761+0000 7f5c10ba8640 1 -- 172.21.15.146:0/1141112740 wait complete. 2024-04-25T15:23:23.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T15:23:23.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.880+0000 7feb1b815640 1 Processor -- start 2024-04-25T15:23:23.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.880+0000 7feb1b815640 1 -- start start 2024-04-25T15:23:23.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb1b815640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7feb14153ca0 0x7feb1415dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb1b815640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb140cf420 con 0x7feb140cbc90 2024-04-25T15:23:23.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb1b815640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb140cc270 con 0x7feb140d0ac0 2024-04-25T15:23:23.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb1b815640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7feb140cfd10 con 0x7feb14153ca0 2024-04-25T15:23:23.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb195b1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7feb140cbc90 0x7feb140d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43558/0 (socket says 172.21.15.146:43558) 2024-04-25T15:23:23.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb195b1640 1 -- 172.21.15.146:0/2105037604 learned_addr learned my addr 172.21.15.146:0/2105037604 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb18db0640 1 --2- 172.21.15.146:0/2105037604 >> v2:172.21.15.177:3300/0 conn(0x7feb14153ca0 0x7feb1415dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427262561 0 0) 0x7feb140cf420 con 0x7feb140cbc90 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb00003520 con 0x7feb140cbc90 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3919613776 0 0) 0x7feb00003520 con 0x7feb140cbc90 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 >> v2:172.21.15.177:3300/0 conn(0x7feb14153ca0 msgr2=0x7feb1415dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb0bfff640 1 --2- 172.21.15.146:0/2105037604 >> v2:172.21.15.177:3300/0 conn(0x7feb14153ca0 0x7feb1415dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.882+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 >> v1:172.21.15.177:6789/0 conn(0x7feb140d0ac0 legacy=0x7feb140d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb140cf6c0 con 0x7feb140cbc90 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 839053657 0 0) 0x7feb10001670 con 0x7feb140cbc90 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 >> v1:172.21.15.146:6789/0 conn(0x7feb140cbc90 legacy=0x7feb140d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb000039b0 0x7feb00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00004800 0x7feb00004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb000051c0 0x7feb0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feb140cfd10 con 0x7feb000051c0 2024-04-25T15:23:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb0000c160 con 0x7feb00004800 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7feb0000c360 con 0x7feb000039b0 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb19db2640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb000051c0 0x7feb0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb195b1640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb000039b0 0x7feb00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb18db0640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00004800 0x7feb00004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb19db2640 1 -- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb000039b0 msgr2=0x7feb00003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb19db2640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb000039b0 0x7feb00003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb19db2640 1 -- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00004800 msgr2=0x7feb00004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb19db2640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00004800 0x7feb00004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb19db2640 1 -- 172.21.15.146:0/2105037604 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb00003520 con 0x7feb000051c0 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.883+0000 7feb18db0640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb00004800 0x7feb00004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.884+0000 7feb19db2640 1 -- 172.21.15.146:0/2105037604 reap_dead start 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.884+0000 7feb19db2640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb000051c0 0x7feb0000b9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7feb0c001e20 tx=0x7feb00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465e90693b3cc821 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.884+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c001a70 con 0x7feb000051c0 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.884+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c02ad70 con 0x7feb000051c0 2024-04-25T15:23:23.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.884+0000 7feb0bfff640 1 -- 172.21.15.146:0/2105037604 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c003bb0 con 0x7feb000051c0 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 -- 172.21.15.146:0/2105037604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb000051c0 msgr2=0x7feb0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb000051c0 0x7feb0000b9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7feb0c001e20 tx=0x7feb00005830 comp rx=0 tx=0).stop 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 -- 172.21.15.146:0/2105037604 shutdown_connections 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 --2- 172.21.15.146:0/2105037604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb000051c0 0x7feb0000b9c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 -- 172.21.15.146:0/2105037604 >> 172.21.15.146:0/2105037604 conn(0x7feb140ce0c0 msgr2=0x7feb140ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 -- 172.21.15.146:0/2105037604 shutdown_connections 2024-04-25T15:23:23.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.885+0000 7feb1b815640 1 -- 172.21.15.146:0/2105037604 wait complete. 2024-04-25T15:23:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.886+0000 7feb1b815640 1 Processor -- start 2024-04-25T15:23:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.887+0000 7feb1b815640 1 -- start start 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb1b815640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000051c0 0x7feb14157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb1b815640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb14142b40 0x7feb141443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb1b815640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 0x7feb14154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb1b815640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feb140d01a0 con 0x7feb14156980 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb1b815640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb141540b0 con 0x7feb000051c0 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb1b815640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7feb140d0320 con 0x7feb14142b40 2024-04-25T15:23:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 0x7feb14154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 0x7feb14154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57092/0 (socket says 172.21.15.146:57092) 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 -- 172.21.15.146:0/911834993 learned_addr learned my addr 172.21.15.146:0/911834993 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb18db0640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb14142b40 0x7feb141443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb195b1640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000051c0 0x7feb14157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 -- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb14142b40 msgr2=0x7feb141443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb14142b40 0x7feb141443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 -- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000051c0 msgr2=0x7feb14157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:23.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000051c0 0x7feb14157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 -- 172.21.15.146:0/911834993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb14144930 con 0x7feb14156980 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb19db2640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 0x7feb14154800 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7feb0c001e80 tx=0x7feb0c001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c006d60 con 0x7feb14156980 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.888+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c005500 con 0x7feb14156980 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.889+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb14143350 con 0x7feb14156980 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.889+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c007030 con 0x7feb14156980 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.889+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb14143910 con 0x7feb14156980 2024-04-25T15:23:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.890+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb1406a0e0 con 0x7feb14156980 2024-04-25T15:23:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.890+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7feb0c0071d0 con 0x7feb14156980 2024-04-25T15:23:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.891+0000 7feb09ffb640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feaf8074600 0x7feaf8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.891+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7feb0c0b4c10 con 0x7feb14156980 2024-04-25T15:23:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.891+0000 7feb195b1640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feaf8074600 0x7feaf8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.891+0000 7feb195b1640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feaf8074600 0x7feaf8076b10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feb1406c600 tx=0x7feb10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:23.895+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb0c07f490 con 0x7feb14156980 2024-04-25T15:23:24.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.075+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7feb140711c0 con 0x7feaf8074600 2024-04-25T15:23:24.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.076+0000 7feb09ffb640 1 -- 172.21.15.146:0/911834993 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7feb140711c0 con 0x7feaf8074600 2024-04-25T15:23:24.077 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feaf8074600 msgr2=0x7feaf8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feaf8074600 0x7feaf8076b10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feb1406c600 tx=0x7feb10009040 comp rx=0 tx=0).stop 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 msgr2=0x7feb14154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 0x7feb14154800 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7feb0c001e80 tx=0x7feb0c001510 comp rx=0 tx=0).stop 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 shutdown_connections 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7feaf8074600 0x7feaf8076b10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb14156980 0x7feb14154800 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb14142b40 0x7feb141443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 --2- 172.21.15.146:0/911834993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb000051c0 0x7feb14157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 >> 172.21.15.146:0/911834993 conn(0x7feb140ce0c0 msgr2=0x7feb140af240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 shutdown_connections 2024-04-25T15:23:24.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.078+0000 7feb1b815640 1 -- 172.21.15.146:0/911834993 wait complete. 2024-04-25T15:23:24.090 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T15:23:24.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.201+0000 7f89ad896640 1 Processor -- start 2024-04-25T15:23:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.202+0000 7f89ad896640 1 -- start start 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89ad896640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89a8162590 0x7f89a80b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89ad896640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f89a80d1ff0 0x7f89a80d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89ad896640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a815f070 con 0x7f89a80b52b0 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89ad896640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f89a8072fe0 con 0x7f89a8162590 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89ad896640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f89a81629a0 con 0x7f89a80d1ff0 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89a8162590 0x7f89a80b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f89a8162590 0x7f89a80b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57094/0 (socket says 172.21.15.146:57094) 2024-04-25T15:23:24.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a67fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f89a80b52b0 0x7f89a80b56c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43574/0 (socket says 172.21.15.146:43574) 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 -- 172.21.15.146:0/3563774239 learned_addr learned my addr 172.21.15.146:0/3563774239 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a77fe640 1 --2- 172.21.15.146:0/3563774239 >> v2:172.21.15.177:3300/0 conn(0x7f89a80d1ff0 0x7f89a80d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 -- 172.21.15.146:0/3563774239 >> v2:172.21.15.177:3300/0 conn(0x7f89a80d1ff0 msgr2=0x7f89a80d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 --2- 172.21.15.146:0/3563774239 >> v2:172.21.15.177:3300/0 conn(0x7f89a80d1ff0 0x7f89a80d2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 -- 172.21.15.146:0/3563774239 >> v1:172.21.15.146:6789/0 conn(0x7f89a80b52b0 legacy=0x7f89a80b56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a5ffb640 1 -- 172.21.15.146:0/3563774239 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4229341701 0 0) 0x7f89a815f070 con 0x7f89a80b52b0 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a6ffd640 1 -- 172.21.15.146:0/3563774239 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a815f350 con 0x7f89a8162590 2024-04-25T15:23:24.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.203+0000 7f89a77fe640 1 --2- 172.21.15.146:0/3563774239 >> v2:172.21.15.177:3300/0 conn(0x7f89a80d1ff0 0x7f89a80d2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:24.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.204+0000 7f89a6ffd640 1 --2- 172.21.15.146:0/3563774239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a80b2c10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f89a80d1660 tx=0x7f899802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03d0b2dae60459e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.204+0000 7f89a5ffb640 1 -- 172.21.15.146:0/3563774239 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998036070 con 0x7f89a8162590 2024-04-25T15:23:24.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.204+0000 7f89a5ffb640 1 -- 172.21.15.146:0/3563774239 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8998031600 con 0x7f89a8162590 2024-04-25T15:23:24.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.204+0000 7f89a5ffb640 1 -- 172.21.15.146:0/3563774239 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89980319d0 con 0x7f89a8162590 2024-04-25T15:23:24.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 -- 172.21.15.146:0/3563774239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 msgr2=0x7f89a80b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 --2- 172.21.15.146:0/3563774239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a80b2c10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f89a80d1660 tx=0x7f899802a730 comp rx=0 tx=0).stop 2024-04-25T15:23:24.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 -- 172.21.15.146:0/3563774239 shutdown_connections 2024-04-25T15:23:24.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 --2- 172.21.15.146:0/3563774239 >> v2:172.21.15.177:3300/0 conn(0x7f89a80d1ff0 0x7f89a80d2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 --2- 172.21.15.146:0/3563774239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a80b2c10 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 -- 172.21.15.146:0/3563774239 >> 172.21.15.146:0/3563774239 conn(0x7f89a80d02b0 msgr2=0x7f89a80d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 -- 172.21.15.146:0/3563774239 shutdown_connections 2024-04-25T15:23:24.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.205+0000 7f89ad896640 1 -- 172.21.15.146:0/3563774239 wait complete. 2024-04-25T15:23:24.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.207+0000 7f89ad896640 1 Processor -- start 2024-04-25T15:23:24.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.207+0000 7f89ad896640 1 -- start start 2024-04-25T15:23:24.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89ad896640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89a80b52b0 0x7f89a80c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89ad896640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89a80d1ff0 0x7f89a80c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89ad896640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89ad896640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8162070 con 0x7f89a8162590 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89ad896640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8161d90 con 0x7f89a80d1ff0 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89ad896640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f89a80d2940 con 0x7f89a80b52b0 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89a77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89a77fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a815cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57110/0 (socket says 172.21.15.146:57110) 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89a77fe640 1 -- 172.21.15.146:0/578102 learned_addr learned my addr 172.21.15.146:0/578102 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89a6ffd640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89a80b52b0 0x7f89a80c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.208+0000 7f89a67fc640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89a80d1ff0 0x7f89a80c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89a77fe640 1 -- 172.21.15.146:0/578102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89a80b52b0 msgr2=0x7f89a80c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89a77fe640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f89a80b52b0 0x7f89a80c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89a77fe640 1 -- 172.21.15.146:0/578102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89a80d1ff0 msgr2=0x7f89a80c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89a77fe640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89a80d1ff0 0x7f89a80c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89a77fe640 1 -- 172.21.15.146:0/578102 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a80c8150 con 0x7f89a8162590 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89a77fe640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a815cf10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f89a000c7f0 tx=0x7f89a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0015020 con 0x7f89a8162590 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a0004230 con 0x7f89a8162590 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a80c8430 con 0x7f89a8162590 2024-04-25T15:23:24.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0007620 con 0x7f89a8162590 2024-04-25T15:23:24.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.209+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a80c89f0 con 0x7f89a8162590 2024-04-25T15:23:24.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.210+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89a806a0e0 con 0x7f89a8162590 2024-04-25T15:23:24.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f89a0018020 con 0x7f89a8162590 2024-04-25T15:23:24.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f898ffff640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89940745d0 0x7f8994076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f89a6ffd640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89940745d0 0x7f8994076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f89a0095ac0 con 0x7f89a8162590 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f898ffff640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f899407ab60 0x7f899407cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 --> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f899407d6c0 con 0x7f899407ab60 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.211+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f89a0095f50 con 0x7f89a8162590 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.212+0000 7f89a6ffd640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89940745d0 0x7f8994076ae0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8998033000 tx=0x7f8998009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.212+0000 7f89a67fc640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f899407ab60 0x7f899407cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.212+0000 7f89a67fc640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f899407ab60 0x7f899407cff0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.213+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== osd.4 v2:172.21.15.177:6817/761443217 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f899407d6c0 con 0x7f899407ab60 2024-04-25T15:23:24.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.226+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 --> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f89a815b660 con 0x7f899407ab60 2024-04-25T15:23:24.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f898ffff640 1 -- 172.21.15.146:0/578102 <== osd.4 v2:172.21.15.177:6817/761443217 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f899c0072e0 con 0x7f899407ab60 2024-04-25T15:23:24.227 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:24.228 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:24.228 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f899407ab60 msgr2=0x7f899407cff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.177:6817/761443217,v1:172.21.15.177:6819/761443217] conn(0x7f899407ab60 0x7f899407cff0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89940745d0 msgr2=0x7f8994076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f89940745d0 0x7f8994076ae0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8998033000 tx=0x7f8998009000 comp rx=0 tx=0).stop 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 msgr2=0x7f89a815cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 --2- 172.21.15.146:0/578102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89a8162590 0x7f89a815cf10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f89a000c7f0 tx=0x7f89a000ccb0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89a77fe640 1 -- 172.21.15.146:0/578102 reap_dead start 2024-04-25T15:23:24.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 shutdown_connections 2024-04-25T15:23:24.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 >> 172.21.15.146:0/578102 conn(0x7f89a80d02b0 msgr2=0x7f89a815e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 shutdown_connections 2024-04-25T15:23:24.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.227+0000 7f89ad896640 1 -- 172.21.15.146:0/578102 wait complete. 2024-04-25T15:23:24.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T15:23:24.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.346+0000 7f6fb2084640 1 Processor -- start 2024-04-25T15:23:24.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.347+0000 7f6fb2084640 1 -- start start 2024-04-25T15:23:24.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fb2084640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac0715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fb2084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fac071bc0 con 0x7f6fac070360 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fb2084640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fac0d1b00 con 0x7f6fac160000 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fb2084640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6fac0d1de0 con 0x7f6fac0b7350 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fabfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6fac070360 0x7f6fac169ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43586/0 (socket says 172.21.15.146:43586) 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fabfff640 1 -- 172.21.15.146:0/1780040555 learned_addr learned my addr 172.21.15.146:0/1780040555 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6fab7fe640 1 --2- 172.21.15.146:0/1780040555 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac0715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354541863 0 0) 0x7f6fac071bc0 con 0x7f6fac070360 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.348+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f94003520 con 0x7f6fac070360 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2079440445 0 0) 0x7f6fac0d1b00 con 0x7f6fac160000 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fac071bc0 con 0x7f6fac160000 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6fab7fe640 1 -- 172.21.15.146:0/1780040555 >> v1:172.21.15.177:6789/0 conn(0x7f6fac160000 legacy=0x7f6fac06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6fab7fe640 1 -- 172.21.15.146:0/1780040555 >> v1:172.21.15.146:6789/0 conn(0x7f6fac070360 legacy=0x7f6fac169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4067990292 0 0) 0x7f6f94003520 con 0x7f6fac070360 2024-04-25T15:23:24.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6fab7fe640 1 -- 172.21.15.146:0/1780040555 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac0d20c0 con 0x7f6fac0b7350 2024-04-25T15:23:24.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.349+0000 7f6fab7fe640 1 --2- 172.21.15.146:0/1780040555 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac0715e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0097d0 tx=0x7f6f9c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8e8434ecd9a7ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c037070 con 0x7f6fac0b7350 2024-04-25T15:23:24.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9c02ab90 con 0x7f6fac0b7350 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6faa7fc640 1 -- 172.21.15.146:0/1780040555 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40019f0 con 0x7f6fac0b7350 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 -- 172.21.15.146:0/1780040555 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 msgr2=0x7f6fac0715e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 --2- 172.21.15.146:0/1780040555 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac0715e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0097d0 tx=0x7f6f9c02a100 comp rx=0 tx=0).stop 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 -- 172.21.15.146:0/1780040555 shutdown_connections 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 --2- 172.21.15.146:0/1780040555 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac0715e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 -- 172.21.15.146:0/1780040555 >> 172.21.15.146:0/1780040555 conn(0x7f6fac09b2b0 msgr2=0x7f6fac0b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 -- 172.21.15.146:0/1780040555 shutdown_connections 2024-04-25T15:23:24.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.350+0000 7f6fb2084640 1 -- 172.21.15.146:0/1780040555 wait complete. 2024-04-25T15:23:24.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.352+0000 7f6fb2084640 1 Processor -- start 2024-04-25T15:23:24.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.352+0000 7f6fb2084640 1 -- start start 2024-04-25T15:23:24.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.353+0000 7f6fb2084640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.353+0000 7f6fb2084640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 0x7f6fac1f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fb2084640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac1490b0 0x7f6fac1494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fb2084640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac070980 con 0x7f6fac160000 2024-04-25T15:23:24.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fb2084640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac0cf4b0 con 0x7f6fac1490b0 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fb2084640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f6fac160410 con 0x7f6fac0b7350 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 0x7f6fac1f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 0x7f6fac1f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57122/0 (socket says 172.21.15.146:57122) 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 -- 172.21.15.146:0/3322949272 learned_addr learned my addr 172.21.15.146:0/3322949272 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fab7fe640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fabfff640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac1490b0 0x7f6fac1494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 -- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 msgr2=0x7f6fac159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 -- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac1490b0 msgr2=0x7f6fac1494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac1490b0 0x7f6fac1494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 -- 172.21.15.146:0/3322949272 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac149a00 con 0x7f6fac160000 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fab7fe640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6faaffd640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 0x7f6fac1f59d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c3c0 tx=0x7f6fa0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0017070 con 0x7f6fac160000 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa00052d0 con 0x7f6fac160000 2024-04-25T15:23:24.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac15c210 con 0x7f6fac160000 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0003830 con 0x7f6fac160000 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.354+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac15c7b0 con 0x7f6fac160000 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.355+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fac06a0e0 con 0x7f6fac160000 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.356+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f6fa001a070 con 0x7f6fac160000 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.357+0000 7f6f93fff640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f980745d0 0x7f6f98076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.357+0000 7f6fab7fe640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f980745d0 0x7f6f98076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.357+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f6fa00946e0 con 0x7f6fac160000 2024-04-25T15:23:24.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.357+0000 7f6fab7fe640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f980745d0 0x7f6f98076ae0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c034000 tx=0x7f6f9c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.361+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fac149a00 con 0x7f6fac160000 2024-04-25T15:23:24.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.535+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f6fac0b77f0 con 0x7f6f980745d0 2024-04-25T15:23:24.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.535+0000 7f6f93fff640 1 -- 172.21.15.146:0/3322949272 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6fac0b77f0 con 0x7f6f980745d0 2024-04-25T15:23:24.536 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T15:23:24.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f980745d0 msgr2=0x7f6f98076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f980745d0 0x7f6f98076ae0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c034000 tx=0x7f6f9c02e000 comp rx=0 tx=0).stop 2024-04-25T15:23:24.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 msgr2=0x7f6fac1f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 0x7f6fac1f59d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c3c0 tx=0x7f6fa0001e70 comp rx=0 tx=0).stop 2024-04-25T15:23:24.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 shutdown_connections 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f6f980745d0 0x7f6f98076ae0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac1490b0 0x7f6fac1494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fac160000 0x7f6fac1f59d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 --2- 172.21.15.146:0/3322949272 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fac0b7350 0x7f6fac159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 >> 172.21.15.146:0/3322949272 conn(0x7f6fac09b2b0 msgr2=0x7f6fac15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 shutdown_connections 2024-04-25T15:23:24.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.537+0000 7f6fb2084640 1 -- 172.21.15.146:0/3322949272 wait complete. 2024-04-25T15:23:24.549 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T15:23:24.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:24.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.661+0000 7f133391f640 1 Processor -- start 2024-04-25T15:23:24.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.661+0000 7f133391f640 1 -- start start 2024-04-25T15:23:24.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f133391f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f132c0d0220 0x7f132c0ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f133391f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0cdcb0 con 0x7f132c15ce50 2024-04-25T15:23:24.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f133391f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0cbe70 con 0x7f132c0cd7d0 2024-04-25T15:23:24.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f133391f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f132c0cbc20 con 0x7f132c0d0220 2024-04-25T15:23:24.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f1330eba640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f132c0d0220 0x7f132c0ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f1331ebc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f132c15ce50 0x7f132c070750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43592/0 (socket says 172.21.15.146:43592) 2024-04-25T15:23:24.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f1331ebc640 1 -- 172.21.15.146:0/2164646891 learned_addr learned my addr 172.21.15.146:0/2164646891 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:24.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f1330eba640 1 -- 172.21.15.146:0/2164646891 >> v1:172.21.15.177:6789/0 conn(0x7f132c0cd7d0 legacy=0x7f132c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f1330eba640 1 -- 172.21.15.146:0/2164646891 >> v1:172.21.15.146:6789/0 conn(0x7f132c15ce50 legacy=0x7f132c070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.663+0000 7f1330eba640 1 -- 172.21.15.146:0/2164646891 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0cc150 con 0x7f132c0d0220 2024-04-25T15:23:24.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.664+0000 7f1330eba640 1 --2- 172.21.15.146:0/2164646891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c0ae5f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f131c0097a0 tx=0x7f131c02c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f695e664fc3e1744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.664+0000 7f1323fff640 1 -- 172.21.15.146:0/2164646891 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c039070 con 0x7f132c0d0220 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.664+0000 7f1323fff640 1 -- 172.21.15.146:0/2164646891 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c034640 con 0x7f132c0d0220 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.664+0000 7f1323fff640 1 -- 172.21.15.146:0/2164646891 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c034a10 con 0x7f132c0d0220 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 -- 172.21.15.146:0/2164646891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 msgr2=0x7f132c0ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 --2- 172.21.15.146:0/2164646891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c0ae5f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f131c0097a0 tx=0x7f131c02c320 comp rx=0 tx=0).stop 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 -- 172.21.15.146:0/2164646891 shutdown_connections 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 --2- 172.21.15.146:0/2164646891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c0ae5f0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 -- 172.21.15.146:0/2164646891 >> 172.21.15.146:0/2164646891 conn(0x7f132c0ce470 msgr2=0x7f132c0ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 -- 172.21.15.146:0/2164646891 shutdown_connections 2024-04-25T15:23:24.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.665+0000 7f133391f640 1 -- 172.21.15.146:0/2164646891 wait complete. 2024-04-25T15:23:24.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.667+0000 7f133391f640 1 Processor -- start 2024-04-25T15:23:24.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.667+0000 7f133391f640 1 -- start start 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f133391f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132c0cd7d0 0x7f132c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f133391f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f133391f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f132c15ce50 0x7f132c0b5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f133391f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f132c06d230 con 0x7f132c0d0220 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f133391f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f132c06d0b0 con 0x7f132c0cd7d0 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f133391f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f132c06d3b0 con 0x7f132c15ce50 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c1500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57134/0 (socket says 172.21.15.146:57134) 2024-04-25T15:23:24.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 -- 172.21.15.146:0/431372414 learned_addr learned my addr 172.21.15.146:0/431372414 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f13316bb640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132c0cd7d0 0x7f132c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1331ebc640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f132c15ce50 0x7f132c0b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 -- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f132c15ce50 msgr2=0x7f132c0b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f132c15ce50 0x7f132c0b5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 -- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132c0cd7d0 msgr2=0x7f132c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f132c0cd7d0 0x7f132c15d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1330eba640 1 -- 172.21.15.146:0/431372414 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c1f2db0 con 0x7f132c0d0220 2024-04-25T15:23:24.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.668+0000 7f1331ebc640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f132c15ce50 0x7f132c0b5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.669+0000 7f1330eba640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c1500b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f131c037040 tx=0x7f131c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.669+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c043030 con 0x7f132c0d0220 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.669+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c032750 con 0x7f132c0d0220 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.669+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c1f3090 con 0x7f132c0d0220 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.669+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c039040 con 0x7f132c0d0220 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.669+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1f3650 con 0x7f132c0d0220 2024-04-25T15:23:24.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.670+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f132c06a0e0 con 0x7f132c0d0220 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f131c046070 con 0x7f132c0d0220 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1321ffb640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13100745d0 0x7f1310076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f131c0b8470 con 0x7f132c0d0220 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f13316bb640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13100745d0 0x7f1310076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1321ffb640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7f131007ab60 0x7f131007cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 --> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f131007d6c0 con 0x7f131007ab60 2024-04-25T15:23:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f131c0b8820 con 0x7f132c0d0220 2024-04-25T15:23:24.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.671+0000 7f1331ebc640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7f131007ab60 0x7f131007cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.672+0000 7f13316bb640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13100745d0 0x7f1310076ae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1328001e70 tx=0x7f1328009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.672+0000 7f1331ebc640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7f131007ab60 0x7f131007cff0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.673+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== osd.5 v2:172.21.15.177:6808/2540256124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f131007d6c0 con 0x7f131007ab60 2024-04-25T15:23:24.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.686+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 --> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f132c06c860 con 0x7f131007ab60 2024-04-25T15:23:24.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.686+0000 7f1321ffb640 1 -- 172.21.15.146:0/431372414 <== osd.5 v2:172.21.15.177:6808/2540256124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f132c06c860 con 0x7f131007ab60 2024-04-25T15:23:24.687 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:24.687 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:24.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7f131007ab60 msgr2=0x7f131007cff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.177:6808/2540256124,v1:172.21.15.177:6809/2540256124] conn(0x7f131007ab60 0x7f131007cff0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13100745d0 msgr2=0x7f1310076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f13100745d0 0x7f1310076ae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1328001e70 tx=0x7f1328009040 comp rx=0 tx=0).stop 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 msgr2=0x7f132c1500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 --2- 172.21.15.146:0/431372414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f132c0d0220 0x7f132c1500b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f131c037040 tx=0x7f131c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f1331ebc640 1 -- 172.21.15.146:0/431372414 reap_dead start 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 shutdown_connections 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 >> 172.21.15.146:0/431372414 conn(0x7f132c0ce470 msgr2=0x7f132c0b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 shutdown_connections 2024-04-25T15:23:24.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.687+0000 7f133391f640 1 -- 172.21.15.146:0/431372414 wait complete. 2024-04-25T15:23:24.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T15:23:24.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.807+0000 7feebd8b1640 1 Processor -- start 2024-04-25T15:23:24.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.807+0000 7feebd8b1640 1 -- start start 2024-04-25T15:23:24.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feebd8b1640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7feeb80d4f60 0x7feeb80d5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feebd8b1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feeb80d0020 0x7feeb80d0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feebd8b1640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb80b9200 con 0x7feeb8163d80 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feebd8b1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7feeb806c550 con 0x7feeb80d0020 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feebd8b1640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7feeb806c830 con 0x7feeb80d4f60 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feeae7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feeb80d0020 0x7feeb80d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feeae7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feeb80d0020 0x7feeb80d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57140/0 (socket says 172.21.15.146:57140) 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feeae7fc640 1 -- 172.21.15.146:0/1726990230 learned_addr learned my addr 172.21.15.146:0/1726990230 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.808+0000 7feeb6ffd640 1 --2- 172.21.15.146:0/1726990230 >> v2:172.21.15.177:3300/0 conn(0x7feeb80d4f60 0x7feeb80d5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.809+0000 7feeae7fc640 1 -- 172.21.15.146:0/1726990230 >> v2:172.21.15.177:3300/0 conn(0x7feeb80d4f60 msgr2=0x7feeb80d5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:24.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.809+0000 7feeae7fc640 1 --2- 172.21.15.146:0/1726990230 >> v2:172.21.15.177:3300/0 conn(0x7feeb80d4f60 0x7feeb80d5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.809+0000 7feeae7fc640 1 -- 172.21.15.146:0/1726990230 >> v1:172.21.15.177:6789/0 conn(0x7feeb8163d80 legacy=0x7feeb8164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.809+0000 7feeae7fc640 1 -- 172.21.15.146:0/1726990230 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb80d0970 con 0x7feeb80d0020 2024-04-25T15:23:24.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.809+0000 7feeae7fc640 1 --2- 172.21.15.146:0/1726990230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb80d0430 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7feeb80d5930 tx=0x7feea002a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9016bca7c91edfe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.809+0000 7feeb67fc640 1 -- 172.21.15.146:0/1726990230 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0037070 con 0x7feeb80d0020 2024-04-25T15:23:24.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feeb67fc640 1 -- 172.21.15.146:0/1726990230 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea0032440 con 0x7feeb80d0020 2024-04-25T15:23:24.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feeb67fc640 1 -- 172.21.15.146:0/1726990230 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0032810 con 0x7feeb80d0020 2024-04-25T15:23:24.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 -- 172.21.15.146:0/1726990230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 msgr2=0x7feeb80d0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1726990230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb80d0430 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7feeb80d5930 tx=0x7feea002a150 comp rx=0 tx=0).stop 2024-04-25T15:23:24.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 -- 172.21.15.146:0/1726990230 shutdown_connections 2024-04-25T15:23:24.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1726990230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb80d0430 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1726990230 >> v2:172.21.15.177:3300/0 conn(0x7feeb80d4f60 0x7feeb80d5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 -- 172.21.15.146:0/1726990230 >> 172.21.15.146:0/1726990230 conn(0x7feeb80d2450 msgr2=0x7feeb80d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:24.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 -- 172.21.15.146:0/1726990230 shutdown_connections 2024-04-25T15:23:24.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.810+0000 7feebd8b1640 1 -- 172.21.15.146:0/1726990230 wait complete. 2024-04-25T15:23:24.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.812+0000 7feebd8b1640 1 Processor -- start 2024-04-25T15:23:24.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.812+0000 7feebd8b1640 1 -- start start 2024-04-25T15:23:24.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.813+0000 7feebd8b1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb81eb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.813+0000 7feebd8b1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb80d4f60 0x7feeb815bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feebd8b1640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feeb8163d80 0x7feeb815f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feebd8b1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feeb80d15f0 con 0x7feeb80d0020 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feebd8b1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feeb80d1070 con 0x7feeb80d4f60 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feebd8b1640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7feeb80b5150 con 0x7feeb8163d80 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb81eb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb81eb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57152/0 (socket says 172.21.15.146:57152) 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 -- 172.21.15.146:0/1932753695 learned_addr learned my addr 172.21.15.146:0/1932753695 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 -- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feeb8163d80 msgr2=0x7feeb815f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb77fe640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feeb8163d80 0x7feeb815f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeae7fc640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb80d4f60 0x7feeb815bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feeb8163d80 0x7feeb815f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 -- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb80d4f60 msgr2=0x7feeb815bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb80d4f60 0x7feeb815bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 -- 172.21.15.146:0/1932753695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb815d580 con 0x7feeb80d0020 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb77fe640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feeb8163d80 0x7feeb815f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeae7fc640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb80d4f60 0x7feeb815bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb6ffd640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb81eb050 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7feea800e8a0 tx=0x7feea800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8017070 con 0x7feeb80d0020 2024-04-25T15:23:24.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea800ee90 con 0x7feeb80d0020 2024-04-25T15:23:24.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb815d820 con 0x7feeb80d0020 2024-04-25T15:23:24.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea800cb00 con 0x7feeb80d0020 2024-04-25T15:23:24.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.814+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb815dde0 con 0x7feeb80d0020 2024-04-25T15:23:24.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.815+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb806a0e0 con 0x7feeb80d0020 2024-04-25T15:23:24.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.816+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7feea801a070 con 0x7feeb80d0020 2024-04-25T15:23:24.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.817+0000 7feeb4ff9640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee980745d0 0x7fee98076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:24.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.817+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7feea80950e0 con 0x7feeb80d0020 2024-04-25T15:23:24.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.817+0000 7feeae7fc640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee980745d0 0x7fee98076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:24.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.817+0000 7feeae7fc640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee980745d0 0x7fee98076ae0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7feea0034000 tx=0x7feea002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:24.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.821+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feea805f880 con 0x7feeb80d0020 2024-04-25T15:23:25.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.999+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7feeb81639b0 con 0x7fee980745d0 2024-04-25T15:23:25.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:24.999+0000 7feeb4ff9640 1 -- 172.21.15.146:0/1932753695 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7feeb81639b0 con 0x7fee980745d0 2024-04-25T15:23:25.001 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T15:23:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.001+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee980745d0 msgr2=0x7fee98076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.001+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee980745d0 0x7fee98076ae0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7feea0034000 tx=0x7feea002e000 comp rx=0 tx=0).stop 2024-04-25T15:23:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 msgr2=0x7feeb81eb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb81eb050 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7feea800e8a0 tx=0x7feea800ed60 comp rx=0 tx=0).stop 2024-04-25T15:23:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 shutdown_connections 2024-04-25T15:23:25.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fee980745d0 0x7fee98076ae0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feeb8163d80 0x7feeb815f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb80d4f60 0x7feeb815bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 --2- 172.21.15.146:0/1932753695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80d0020 0x7feeb81eb050 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 >> 172.21.15.146:0/1932753695 conn(0x7feeb80d2450 msgr2=0x7feeb81624e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 shutdown_connections 2024-04-25T15:23:25.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.002+0000 7feebd8b1640 1 -- 172.21.15.146:0/1932753695 wait complete. 2024-04-25T15:23:25.014 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T15:23:25.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:25.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.125+0000 7f1e86c1a640 1 Processor -- start 2024-04-25T15:23:25.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.125+0000 7f1e86c1a640 1 -- start start 2024-04-25T15:23:25.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e86c1a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e8006d4f0 0x7f1e801601a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e86c1a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e800cd380 con 0x7f1e80146140 2024-04-25T15:23:25.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e86c1a640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e800cd660 con 0x7f1e80145bd0 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e86c1a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1e800cd940 con 0x7f1e8006d4f0 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e8006d4f0 0x7f1e801601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e8006d4f0 0x7f1e801601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57168/0 (socket says 172.21.15.146:57168) 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e851b7640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1e80146140 0x7f1e8015ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43606/0 (socket says 172.21.15.146:43606) 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7ffff640 1 -- 172.21.15.146:0/643618141 learned_addr learned my addr 172.21.15.146:0/643618141 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7ffff640 1 -- 172.21.15.146:0/643618141 >> v1:172.21.15.177:6789/0 conn(0x7f1e80145bd0 legacy=0x7f1e8006cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7ffff640 1 -- 172.21.15.146:0/643618141 >> v1:172.21.15.146:6789/0 conn(0x7f1e80146140 legacy=0x7f1e8015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7ffff640 1 -- 172.21.15.146:0/643618141 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e800cdc20 con 0x7f1e8006d4f0 2024-04-25T15:23:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.127+0000 7f1e7f7fe640 1 -- 172.21.15.146:0/643618141 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115817997 0 0) 0x7f1e800cd380 con 0x7f1e80146140 2024-04-25T15:23:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.128+0000 7f1e7ffff640 1 --2- 172.21.15.146:0/643618141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e8006d4f0 0x7f1e801601a0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00aa10 tx=0x7f1e6c02b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3134e83450eef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.128+0000 7f1e7f7fe640 1 -- 172.21.15.146:0/643618141 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00b040 con 0x7f1e8006d4f0 2024-04-25T15:23:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.128+0000 7f1e7f7fe640 1 -- 172.21.15.146:0/643618141 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6c02be50 con 0x7f1e8006d4f0 2024-04-25T15:23:25.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.128+0000 7f1e7f7fe640 1 -- 172.21.15.146:0/643618141 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c030930 con 0x7f1e8006d4f0 2024-04-25T15:23:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 -- 172.21.15.146:0/643618141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e8006d4f0 msgr2=0x7f1e801601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 --2- 172.21.15.146:0/643618141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e8006d4f0 0x7f1e801601a0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00aa10 tx=0x7f1e6c02b670 comp rx=0 tx=0).stop 2024-04-25T15:23:25.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 -- 172.21.15.146:0/643618141 shutdown_connections 2024-04-25T15:23:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 --2- 172.21.15.146:0/643618141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e8006d4f0 0x7f1e801601a0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 -- 172.21.15.146:0/643618141 >> 172.21.15.146:0/643618141 conn(0x7f1e800ce470 msgr2=0x7f1e800ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 -- 172.21.15.146:0/643618141 shutdown_connections 2024-04-25T15:23:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.129+0000 7f1e86c1a640 1 -- 172.21.15.146:0/643618141 wait complete. 2024-04-25T15:23:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.130+0000 7f1e86c1a640 1 Processor -- start 2024-04-25T15:23:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.130+0000 7f1e86c1a640 1 -- start start 2024-04-25T15:23:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e86c1a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e80145bd0 0x7f1e800b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e86c1a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1e80146140 0x7f1e80134030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e86c1a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80136cc0 0x7f1e8015c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e86c1a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1e800cd120 con 0x7f1e80136cc0 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e86c1a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e800ccfa0 con 0x7f1e80145bd0 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e86c1a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f1e80071050 con 0x7f1e80146140 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80136cc0 0x7f1e8015c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80136cc0 0x7f1e8015c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57184/0 (socket says 172.21.15.146:57184) 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 -- 172.21.15.146:0/2716759709 learned_addr learned my addr 172.21.15.146:0/2716759709 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e7ffff640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1e80146140 0x7f1e80134030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 -- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1e80146140 msgr2=0x7f1e80134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e849b6640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e80145bd0 0x7f1e800b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1e80146140 0x7f1e80134030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 -- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e80145bd0 msgr2=0x7f1e800b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e80145bd0 0x7f1e800b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 -- 172.21.15.146:0/2716759709 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e801424c0 con 0x7f1e80136cc0 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e7ffff640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1e80146140 0x7f1e80134030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e849b6640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e80145bd0 0x7f1e800b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.132+0000 7f1e851b7640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80136cc0 0x7f1e8015c290 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800c700 tx=0x7f1e7800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.133+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e78018070 con 0x7f1e80136cc0 2024-04-25T15:23:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.133+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e7800ce10 con 0x7f1e80136cc0 2024-04-25T15:23:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.133+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e801427a0 con 0x7f1e80136cc0 2024-04-25T15:23:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.133+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800a880 con 0x7f1e80136cc0 2024-04-25T15:23:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.133+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e80142d60 con 0x7f1e80136cc0 2024-04-25T15:23:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.134+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e8006a0e0 con 0x7f1e80136cc0 2024-04-25T15:23:25.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.134+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f1e7800aa20 con 0x7f1e80136cc0 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e7d7fa640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1e64074600 0x7f1e64076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f1e780946d0 con 0x7f1e80136cc0 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e849b6640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1e64074600 0x7f1e64076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e7d7fa640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f1e6407ab90 0x7f1e6407d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 --> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e6407d6f0 con 0x7f1e6407ab90 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f1e7809b050 con 0x7f1e80136cc0 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e7ffff640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f1e6407ab90 0x7f1e6407d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.135+0000 7f1e849b6640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1e64074600 0x7f1e64076b10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1e801460f0 tx=0x7f1e74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.136+0000 7f1e7ffff640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f1e6407ab90 0x7f1e6407d020 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.136+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== osd.6 v2:172.21.15.177:6800/2938680267 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e6407d6f0 con 0x7f1e6407ab90 2024-04-25T15:23:25.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.149+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 --> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1e800b2fb0 con 0x7f1e6407ab90 2024-04-25T15:23:25.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e7d7fa640 1 -- 172.21.15.146:0/2716759709 <== osd.6 v2:172.21.15.177:6800/2938680267 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1e800b2fb0 con 0x7f1e6407ab90 2024-04-25T15:23:25.151 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:25.151 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:25.151 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:25.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f1e6407ab90 msgr2=0x7f1e6407d020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e86c1a640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.177:6800/2938680267,v1:172.21.15.177:6801/2938680267] conn(0x7f1e6407ab90 0x7f1e6407d020 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1e64074600 msgr2=0x7f1e64076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e86c1a640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f1e64074600 0x7f1e64076b10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1e801460f0 tx=0x7f1e74008040 comp rx=0 tx=0).stop 2024-04-25T15:23:25.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80136cc0 msgr2=0x7f1e8015c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e86c1a640 1 --2- 172.21.15.146:0/2716759709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e80136cc0 0x7f1e8015c290 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800c700 tx=0x7f1e7800cbc0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.150+0000 7f1e851b7640 1 -- 172.21.15.146:0/2716759709 reap_dead start 2024-04-25T15:23:25.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.151+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 shutdown_connections 2024-04-25T15:23:25.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.151+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 >> 172.21.15.146:0/2716759709 conn(0x7f1e800ce470 msgr2=0x7f1e800af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.151+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 shutdown_connections 2024-04-25T15:23:25.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.151+0000 7f1e86c1a640 1 -- 172.21.15.146:0/2716759709 wait complete. 2024-04-25T15:23:25.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T15:23:25.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.269+0000 7fb88cd9d640 1 Processor -- start 2024-04-25T15:23:25.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.269+0000 7fb88cd9d640 1 -- start start 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.270+0000 7fb88cd9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb88806da30 0x7fb88815e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.270+0000 7fb88cd9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb88806cf60 0x7fb88806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb88cd9d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8880b0f60 con 0x7fb8880b0b50 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb88cd9d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb8880b1790 con 0x7fb88806cf60 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb88cd9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb8880b2bb0 con 0x7fb88806da30 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb88806cf60 0x7fb88806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb88806cf60 0x7fb88806d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57198/0 (socket says 172.21.15.146:57198) 2024-04-25T15:23:25.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 -- 172.21.15.146:0/3509700860 learned_addr learned my addr 172.21.15.146:0/3509700860 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 -- 172.21.15.146:0/3509700860 >> v2:172.21.15.177:3300/0 conn(0x7fb88806da30 msgr2=0x7fb88815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 --2- 172.21.15.146:0/3509700860 >> v2:172.21.15.177:3300/0 conn(0x7fb88806da30 0x7fb88815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 -- 172.21.15.146:0/3509700860 >> v1:172.21.15.177:6789/0 conn(0x7fb8880b0b50 legacy=0x7fb8880cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.271+0000 7fb886d9d640 1 -- 172.21.15.146:0/3509700860 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88806def0 con 0x7fb88806cf60 2024-04-25T15:23:25.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.272+0000 7fb886d9d640 1 --2- 172.21.15.146:0/3509700860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb88806cf60 0x7fb88806d370 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fb888073250 tx=0x7fb88002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c14eac9525581b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.272+0000 7fb88559a640 1 -- 172.21.15.146:0/3509700860 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880037070 con 0x7fb88806cf60 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.272+0000 7fb88559a640 1 -- 172.21.15.146:0/3509700860 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb880032440 con 0x7fb88806cf60 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.272+0000 7fb88559a640 1 -- 172.21.15.146:0/3509700860 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880032810 con 0x7fb88806cf60 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.272+0000 7fb88cd9d640 1 -- 172.21.15.146:0/3509700860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb88806cf60 msgr2=0x7fb88806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.272+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/3509700860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb88806cf60 0x7fb88806d370 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fb888073250 tx=0x7fb88002a230 comp rx=0 tx=0).stop 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.273+0000 7fb88cd9d640 1 -- 172.21.15.146:0/3509700860 shutdown_connections 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.273+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/3509700860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb88806cf60 0x7fb88806d370 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.273+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/3509700860 >> v2:172.21.15.177:3300/0 conn(0x7fb88806da30 0x7fb88815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.273+0000 7fb88cd9d640 1 -- 172.21.15.146:0/3509700860 >> 172.21.15.146:0/3509700860 conn(0x7fb88809b2b0 msgr2=0x7fb8880b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.273+0000 7fb88cd9d640 1 -- 172.21.15.146:0/3509700860 shutdown_connections 2024-04-25T15:23:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.273+0000 7fb88cd9d640 1 -- 172.21.15.146:0/3509700860 wait complete. 2024-04-25T15:23:25.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.274+0000 7fb88cd9d640 1 Processor -- start 2024-04-25T15:23:25.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.275+0000 7fb88cd9d640 1 -- start start 2024-04-25T15:23:25.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88cd9d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb88806cf60 0x7fb888143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88cd9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 0x7fb8880cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88cd9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb888146e60 0x7fb888144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88cd9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb8880b2e50 con 0x7fb8880b0b50 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88cd9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8880b14b0 con 0x7fb888146e60 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88cd9d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb8880b3370 con 0x7fb88806cf60 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 0x7fb8880cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 0x7fb8880cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57210/0 (socket says 172.21.15.146:57210) 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 -- 172.21.15.146:0/227770539 learned_addr learned my addr 172.21.15.146:0/227770539 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88659c640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb88806cf60 0x7fb888143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb886d9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb888146e60 0x7fb888144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 -- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb88806cf60 msgr2=0x7fb888143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb88806cf60 0x7fb888143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 -- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb888146e60 msgr2=0x7fb888144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb888146e60 0x7fb888144500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 -- 172.21.15.146:0/227770539 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8880c8ac0 con 0x7fb8880b0b50 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb88659c640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb88806cf60 0x7fb888143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb885d9b640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 0x7fb8880cb2f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00b410 tx=0x7fb87c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.276+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c015020 con 0x7fb8880b0b50 2024-04-25T15:23:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.277+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c011830 con 0x7fb8880b0b50 2024-04-25T15:23:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.277+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8880c8db0 con 0x7fb8880b0b50 2024-04-25T15:23:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.277+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c011c20 con 0x7fb8880b0b50 2024-04-25T15:23:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.277+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8880caaf0 con 0x7fb8880b0b50 2024-04-25T15:23:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.278+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88806a0e0 con 0x7fb8880b0b50 2024-04-25T15:23:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.278+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fb87c018020 con 0x7fb8880b0b50 2024-04-25T15:23:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.279+0000 7fb876ffd640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb8680745d0 0x7fb868076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.279+0000 7fb88659c640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb8680745d0 0x7fb868076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.279+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fb87c0951c0 con 0x7fb8880b0b50 2024-04-25T15:23:25.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.279+0000 7fb88659c640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb8680745d0 0x7fb868076ae0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb8880b2080 tx=0x7fb8780073c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.283+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb87c01f020 con 0x7fb8880b0b50 2024-04-25T15:23:25.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.459+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fb88815c1b0 con 0x7fb8680745d0 2024-04-25T15:23:25.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.459+0000 7fb876ffd640 1 -- 172.21.15.146:0/227770539 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb88815c1b0 con 0x7fb8680745d0 2024-04-25T15:23:25.460 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T15:23:25.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb8680745d0 msgr2=0x7fb868076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb8680745d0 0x7fb868076ae0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb8880b2080 tx=0x7fb8780073c0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 msgr2=0x7fb8880cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 0x7fb8880cb2f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00b410 tx=0x7fb87c00b8d0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 shutdown_connections 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb8680745d0 0x7fb868076ae0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb888146e60 0x7fb888144500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb8880b0b50 0x7fb8880cb2f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 --2- 172.21.15.146:0/227770539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb88806cf60 0x7fb888143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.461+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 >> 172.21.15.146:0/227770539 conn(0x7fb88809b2b0 msgr2=0x7fb888068c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.462+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 shutdown_connections 2024-04-25T15:23:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.462+0000 7fb88cd9d640 1 -- 172.21.15.146:0/227770539 wait complete. 2024-04-25T15:23:25.473 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T15:23:25.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:23:25.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.585+0000 7ffb4b825640 1 Processor -- start 2024-04-25T15:23:25.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.585+0000 7ffb4b825640 1 -- start start 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.586+0000 7ffb4b825640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb440b73e0 0x7ffb4406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.586+0000 7ffb4b825640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440d17c0 0x7ffb440b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.586+0000 7ffb4b825640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb440b3450 con 0x7ffb440b1150 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.586+0000 7ffb4b825640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffb440d2e90 con 0x7ffb440b73e0 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.586+0000 7ffb4b825640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffb440b16a0 con 0x7ffb440d17c0 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb495c1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb440b73e0 0x7ffb4406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb48dc0640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffb440b1150 0x7ffb440d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43612/0 (socket says 172.21.15.146:43612) 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb48dc0640 1 -- 172.21.15.146:0/2341464747 learned_addr learned my addr 172.21.15.146:0/2341464747 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:25.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb49dc2640 1 --2- 172.21.15.146:0/2341464747 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440d17c0 0x7ffb440b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb495c1640 1 -- 172.21.15.146:0/2341464747 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440d17c0 msgr2=0x7ffb440b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:25.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb495c1640 1 --2- 172.21.15.146:0/2341464747 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440d17c0 0x7ffb440b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb495c1640 1 -- 172.21.15.146:0/2341464747 >> v1:172.21.15.146:6789/0 conn(0x7ffb440b1150 legacy=0x7ffb440d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.587+0000 7ffb495c1640 1 -- 172.21.15.146:0/2341464747 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb440d2bb0 con 0x7ffb440b73e0 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb495c1640 1 --2- 172.21.15.146:0/2341464747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440b73e0 0x7ffb4406c550 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ffb440b1820 tx=0x7ffb3c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b22179009ff887 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb3bfff640 1 -- 172.21.15.146:0/2341464747 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c037070 con 0x7ffb440b73e0 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb3bfff640 1 -- 172.21.15.146:0/2341464747 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb3c032440 con 0x7ffb440b73e0 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb3bfff640 1 -- 172.21.15.146:0/2341464747 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c032810 con 0x7ffb440b73e0 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb4b825640 1 -- 172.21.15.146:0/2341464747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440b73e0 msgr2=0x7ffb4406c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb4b825640 1 --2- 172.21.15.146:0/2341464747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440b73e0 0x7ffb4406c550 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ffb440b1820 tx=0x7ffb3c02a180 comp rx=0 tx=0).stop 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb4b825640 1 -- 172.21.15.146:0/2341464747 shutdown_connections 2024-04-25T15:23:25.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb4b825640 1 --2- 172.21.15.146:0/2341464747 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440d17c0 0x7ffb440b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb4b825640 1 --2- 172.21.15.146:0/2341464747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440b73e0 0x7ffb4406c550 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.588+0000 7ffb4b825640 1 -- 172.21.15.146:0/2341464747 >> 172.21.15.146:0/2341464747 conn(0x7ffb4409b2b0 msgr2=0x7ffb440b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.589+0000 7ffb4b825640 1 -- 172.21.15.146:0/2341464747 shutdown_connections 2024-04-25T15:23:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.589+0000 7ffb4b825640 1 -- 172.21.15.146:0/2341464747 wait complete. 2024-04-25T15:23:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.590+0000 7ffb4b825640 1 Processor -- start 2024-04-25T15:23:25.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.590+0000 7ffb4b825640 1 -- start start 2024-04-25T15:23:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440b1150 0x7ffb440c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440b73e0 0x7ffb4409c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440d17c0 0x7ffb4409cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffb44160360 con 0x7ffb440d17c0 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb440d2880 con 0x7ffb440b1150 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7ffb441604e0 con 0x7ffb440b73e0 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440d17c0 0x7ffb4409cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440d17c0 0x7ffb4409cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57220/0 (socket says 172.21.15.146:57220) 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 -- 172.21.15.146:0/59706663 learned_addr learned my addr 172.21.15.146:0/59706663 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb48dc0640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440b73e0 0x7ffb4409c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb495c1640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440b1150 0x7ffb440c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 -- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440b73e0 msgr2=0x7ffb4409c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb440b73e0 0x7ffb4409c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 -- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440b1150 msgr2=0x7ffb440c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440b1150 0x7ffb440c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 -- 172.21.15.146:0/59706663 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb440c6e70 con 0x7ffb440d17c0 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb495c1640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb440b1150 0x7ffb440c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb49dc2640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440d17c0 0x7ffb4409cac0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ffb40007d40 tx=0x7ffb4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40014420 con 0x7ffb440d17c0 2024-04-25T15:23:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb40019070 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb440c7150 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.592+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40014820 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.593+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb4409c060 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.593+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffb4409be80 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.594+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7ffb400149c0 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.594+0000 7ffb39ffb640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb280745d0 0x7ffb28076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.594+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7ffb40095a90 con 0x7ffb440d17c0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb495c1640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb280745d0 0x7ffb28076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb39ffb640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7ffb2807ab60 0x7ffb2807cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 --> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb4000ce50 con 0x7ffb2807ab60 2024-04-25T15:23:25.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7ffb4009a2c0 con 0x7ffb440d17c0 2024-04-25T15:23:25.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb48dc0640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7ffb2807ab60 0x7ffb2807cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb495c1640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb280745d0 0x7ffb28076ae0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c034000 tx=0x7ffb3c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.595+0000 7ffb48dc0640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7ffb2807ab60 0x7ffb2807cff0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.596+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== osd.7 v2:172.21.15.177:6816/1513283347 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffb4000ce50 con 0x7ffb2807ab60 2024-04-25T15:23:25.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.609+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 --> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffb4406a0e0 con 0x7ffb2807ab60 2024-04-25T15:23:25.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb39ffb640 1 -- 172.21.15.146:0/59706663 <== osd.7 v2:172.21.15.177:6816/1513283347 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffb4406a0e0 con 0x7ffb2807ab60 2024-04-25T15:23:25.610 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-25T15:23:25.611 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:23:25.611 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7ffb2807ab60 msgr2=0x7ffb2807cff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.177:6816/1513283347,v1:172.21.15.177:6818/1513283347] conn(0x7ffb2807ab60 0x7ffb2807cff0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb280745d0 msgr2=0x7ffb28076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7ffb280745d0 0x7ffb28076ae0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c034000 tx=0x7ffb3c02e000 comp rx=0 tx=0).stop 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440d17c0 msgr2=0x7ffb4409cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 --2- 172.21.15.146:0/59706663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb440d17c0 0x7ffb4409cac0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ffb40007d40 tx=0x7ffb4000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb49dc2640 1 -- 172.21.15.146:0/59706663 reap_dead start 2024-04-25T15:23:25.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 shutdown_connections 2024-04-25T15:23:25.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 >> 172.21.15.146:0/59706663 conn(0x7ffb4409b2b0 msgr2=0x7ffb44068570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 shutdown_connections 2024-04-25T15:23:25.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.610+0000 7ffb4b825640 1 -- 172.21.15.146:0/59706663 wait complete. 2024-04-25T15:23:25.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T15:23:25.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.732+0000 7fd3da52d640 1 Processor -- start 2024-04-25T15:23:25.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.732+0000 7fd3da52d640 1 -- start start 2024-04-25T15:23:25.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3da52d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40b4cc0 0x7fd3d40b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3da52d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3d40b9430 0x7fd3d40b9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3da52d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d406c8c0 con 0x7fd3d40d42e0 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3da52d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d406d4d0 con 0x7fd3d40b9430 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3da52d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d406cba0 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3d8aca640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3d40b9430 0x7fd3d40b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3d3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd3d40d42e0 0x7fd3d406d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43618/0 (socket says 172.21.15.146:43618) 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3d3fff640 1 -- 172.21.15.146:0/2607826357 learned_addr learned my addr 172.21.15.146:0/2607826357 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3d8aca640 1 --2- 172.21.15.146:0/2607826357 >> v2:172.21.15.146:3300/0 conn(0x7fd3d40b9430 0x7fd3d40b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57232/0 (socket says 172.21.15.146:57232) 2024-04-25T15:23:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.733+0000 7fd3d37fe640 1 --2- 172.21.15.146:0/2607826357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40b4cc0 0x7fd3d40b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d8aca640 1 -- 172.21.15.146:0/2607826357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40b4cc0 msgr2=0x7fd3d40b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d8aca640 1 --2- 172.21.15.146:0/2607826357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40b4cc0 0x7fd3d40b8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d8aca640 1 -- 172.21.15.146:0/2607826357 >> v1:172.21.15.146:6789/0 conn(0x7fd3d40d42e0 legacy=0x7fd3d406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d8aca640 1 -- 172.21.15.146:0/2607826357 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d406ce80 con 0x7fd3d40b9430 2024-04-25T15:23:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d8aca640 1 --2- 172.21.15.146:0/2607826357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b9430 0x7fd3d40b9840 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009d90 tx=0x7fd3cc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dacb792f5dfb6c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d2ffd640 1 -- 172.21.15.146:0/2607826357 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc007740 con 0x7fd3d40b9430 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d2ffd640 1 -- 172.21.15.146:0/2607826357 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc0078e0 con 0x7fd3d40b9430 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.734+0000 7fd3d2ffd640 1 -- 172.21.15.146:0/2607826357 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc007cb0 con 0x7fd3d40b9430 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 -- 172.21.15.146:0/2607826357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b9430 msgr2=0x7fd3d40b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 --2- 172.21.15.146:0/2607826357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b9430 0x7fd3d40b9840 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009d90 tx=0x7fd3cc02a660 comp rx=0 tx=0).stop 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 -- 172.21.15.146:0/2607826357 shutdown_connections 2024-04-25T15:23:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 --2- 172.21.15.146:0/2607826357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b9430 0x7fd3d40b9840 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 --2- 172.21.15.146:0/2607826357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40b4cc0 0x7fd3d40b8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 -- 172.21.15.146:0/2607826357 >> 172.21.15.146:0/2607826357 conn(0x7fd3d40d62b0 msgr2=0x7fd3d40b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 -- 172.21.15.146:0/2607826357 shutdown_connections 2024-04-25T15:23:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.735+0000 7fd3da52d640 1 -- 172.21.15.146:0/2607826357 wait complete. 2024-04-25T15:23:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.737+0000 7fd3da52d640 1 Processor -- start 2024-04-25T15:23:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.737+0000 7fd3da52d640 1 -- start start 2024-04-25T15:23:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.738+0000 7fd3da52d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 0x7fd3d415aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.738+0000 7fd3da52d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3d40b9430 0x7fd3d4160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.738+0000 7fd3da52d640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40d42e0 0x7fd3d40cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.738+0000 7fd3da52d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d406c550 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3da52d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40b50d0 con 0x7fd3d40b9430 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3da52d640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d415f670 con 0x7fd3d40d42e0 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 0x7fd3d415aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 0x7fd3d415aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57244/0 (socket says 172.21.15.146:57244) 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 -- 172.21.15.146:0/426219894 learned_addr learned my addr 172.21.15.146:0/426219894 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:25.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d8aca640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40d42e0 0x7fd3d40cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d37fe640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3d40b9430 0x7fd3d4160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 -- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40d42e0 msgr2=0x7fd3d40cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40d42e0 0x7fd3d40cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 -- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3d40b9430 msgr2=0x7fd3d4160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3d40b9430 0x7fd3d4160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 -- 172.21.15.146:0/426219894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d4162b60 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d3fff640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 0x7fd3d415aac0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800c3c0 tx=0x7fd3c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8017070 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c8005ce0 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c80072f0 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d4162e40 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.739+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d4163400 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.741+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d406a0e0 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.741+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fd3c801a070 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.741+0000 7fd3d0ff9640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3c00745d0 0x7fd3c0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:25.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.742+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fd3c80944e0 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.742+0000 7fd3d37fe640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3c00745d0 0x7fd3c0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:25.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.742+0000 7fd3d37fe640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3c00745d0 0x7fd3c0076ae0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd3d415dfe0 tx=0x7fd3c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:25.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.746+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd3c805e6c0 con 0x7fd3d40b4cc0 2024-04-25T15:23:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.922+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd3d40d0d70 con 0x7fd3c00745d0 2024-04-25T15:23:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.922+0000 7fd3d0ff9640 1 -- 172.21.15.146:0/426219894 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd3d40d0d70 con 0x7fd3c00745d0 2024-04-25T15:23:25.923 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T15:23:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3c00745d0 msgr2=0x7fd3c0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3c00745d0 0x7fd3c0076ae0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd3d415dfe0 tx=0x7fd3c4008040 comp rx=0 tx=0).stop 2024-04-25T15:23:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 msgr2=0x7fd3d415aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 0x7fd3d415aac0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800c3c0 tx=0x7fd3c800c880 comp rx=0 tx=0).stop 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 shutdown_connections 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fd3c00745d0 0x7fd3c0076ae0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3d40d42e0 0x7fd3d40cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3d40b9430 0x7fd3d4160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 --2- 172.21.15.146:0/426219894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40b4cc0 0x7fd3d415aac0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 >> 172.21.15.146:0/426219894 conn(0x7fd3d40d62b0 msgr2=0x7fd3d40b9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 shutdown_connections 2024-04-25T15:23:25.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:25.924+0000 7fd3da52d640 1 -- 172.21.15.146:0/426219894 wait complete. 2024-04-25T15:23:25.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:23:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.044+0000 7f5014738640 1 Processor -- start 2024-04-25T15:23:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.044+0000 7f5014738640 1 -- start start 2024-04-25T15:23:26.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f5014738640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 0x7f500c1622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:26.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f5014738640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f500c0ce020 0x7f500c0b6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:26.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f5014738640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f500c0ce8c0 con 0x7f500c0b5240 2024-04-25T15:23:26.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f5014738640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7f500c0d1950 con 0x7f500c0ce020 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f5014738640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f500c0ceba0 con 0x7f500c15def0 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f50124d4640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 0x7f500c1622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f50124d4640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 0x7f500c1622e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:35594/0 (socket says 172.21.15.146:35594) 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f5011cd3640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f500c0ce020 0x7f500c0b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.046+0000 7f50124d4640 1 -- 172.21.15.146:0/1968368039 learned_addr learned my addr 172.21.15.146:0/1968368039 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.047+0000 7f50124d4640 1 -- 172.21.15.146:0/1968368039 >> v2:172.21.15.177:3300/0 conn(0x7f500c0ce020 msgr2=0x7f500c0b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.047+0000 7f50124d4640 1 --2- 172.21.15.146:0/1968368039 >> v2:172.21.15.177:3300/0 conn(0x7f500c0ce020 0x7f500c0b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.047+0000 7f50124d4640 1 -- 172.21.15.146:0/1968368039 >> v1:172.21.15.177:6789/0 conn(0x7f500c0b5240 legacy=0x7f500c0b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.047+0000 7f50124d4640 1 -- 172.21.15.146:0/1968368039 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500c0cee80 con 0x7f500c15def0 2024-04-25T15:23:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.047+0000 7f5011cd3640 1 --2- 172.21.15.146:0/1968368039 >> v2:172.21.15.177:3300/0 conn(0x7f500c0ce020 0x7f500c0b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.047+0000 7f50124d4640 1 --2- 172.21.15.146:0/1968368039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 0x7f500c1622e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5008009e50 tx=0x7f500802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=165ff07e8fcc98ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f50114d2640 1 -- 172.21.15.146:0/1968368039 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008036070 con 0x7f500c15def0 2024-04-25T15:23:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f50114d2640 1 -- 172.21.15.146:0/1968368039 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5008031600 con 0x7f500c15def0 2024-04-25T15:23:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f50114d2640 1 -- 172.21.15.146:0/1968368039 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50080319d0 con 0x7f500c15def0 2024-04-25T15:23:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 -- 172.21.15.146:0/1968368039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 msgr2=0x7f500c1622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 --2- 172.21.15.146:0/1968368039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 0x7f500c1622e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5008009e50 tx=0x7f500802a710 comp rx=0 tx=0).stop 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 -- 172.21.15.146:0/1968368039 shutdown_connections 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 --2- 172.21.15.146:0/1968368039 >> v2:172.21.15.177:3300/0 conn(0x7f500c0ce020 0x7f500c0b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 --2- 172.21.15.146:0/1968368039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c15def0 0x7f500c1622e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 -- 172.21.15.146:0/1968368039 >> 172.21.15.146:0/1968368039 conn(0x7f500c0d0240 msgr2=0x7f500c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 -- 172.21.15.146:0/1968368039 shutdown_connections 2024-04-25T15:23:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.048+0000 7f5014738640 1 -- 172.21.15.146:0/1968368039 wait complete. 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.050+0000 7f5014738640 1 Processor -- start 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.050+0000 7f5014738640 1 -- start start 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.051+0000 7f5014738640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f500c0b5240 0x7f500c1f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.051+0000 7f5014738640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c0ce020 0x7f500c154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.051+0000 7f5014738640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 0x7f500c070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.051+0000 7f5014738640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f500c0b6ec0 con 0x7f500c15def0 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.051+0000 7f5014738640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f500c0d23a0 con 0x7f500c0b5240 2024-04-25T15:23:26.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.051+0000 7f5014738640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f500c0b7040 con 0x7f500c0ce020 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 0x7f500c070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 0x7f500c070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57246/0 (socket says 172.21.15.146:57246) 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 -- 172.21.15.146:0/3711981803 learned_addr learned my addr 172.21.15.146:0/3711981803 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5011cd3640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c0ce020 0x7f500c154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f50124d4640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f500c0b5240 0x7f500c1f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 -- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c0ce020 msgr2=0x7f500c154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c0ce020 0x7f500c154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 -- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f500c0b5240 msgr2=0x7f500c1f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f500c0b5240 0x7f500c1f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 -- 172.21.15.146:0/3711981803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500c1553b0 con 0x7f500c15def0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5012cd5640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 0x7f500c070100 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f500400c3f0 tx=0x7f500400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004017070 con 0x7f500c15def0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50040052d0 con 0x7f500c15def0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500c155690 con 0x7f500c15def0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004003830 con 0x7f500c15def0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.052+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f500c0cb340 con 0x7f500c15def0 2024-04-25T15:23:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.053+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f500c06a0e0 con 0x7f500c15def0 2024-04-25T15:23:26.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.054+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f500401a070 con 0x7f500c15def0 2024-04-25T15:23:26.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.055+0000 7f5002ffd640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4ff00745d0 0x7f4ff0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.055+0000 7f50124d4640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4ff00745d0 0x7f4ff0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.055+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f50040944c0 con 0x7f500c15def0 2024-04-25T15:23:26.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.055+0000 7f50124d4640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4ff00745d0 0x7f4ff0076ae0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5008033000 tx=0x7f5008009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:26.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.059+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f500405e6d0 con 0x7f500c15def0 2024-04-25T15:23:26.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.230+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f500c0d0900 con 0x7f4ff00745d0 2024-04-25T15:23:26.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.231+0000 7f5002ffd640 1 -- 172.21.15.146:0/3711981803 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13340 (secure 0 0 0) 0x7f500c0d0900 con 0x7f4ff00745d0 2024-04-25T15:23:26.233 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:26.233 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4ff00745d0 msgr2=0x7f4ff0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4ff00745d0 0x7f4ff0076ae0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5008033000 tx=0x7f5008009000 comp rx=0 tx=0).stop 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 msgr2=0x7f500c070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 0x7f500c070100 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f500400c3f0 tx=0x7f500400c7f0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 shutdown_connections 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f4ff00745d0 0x7f4ff0076ae0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f500c15def0 0x7f500c070100 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f500c0ce020 0x7f500c154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 --2- 172.21.15.146:0/3711981803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f500c0b5240 0x7f500c1f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 >> 172.21.15.146:0/3711981803 conn(0x7f500c0d0240 msgr2=0x7f500c0b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 shutdown_connections 2024-04-25T15:23:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:26.233+0000 7f5014738640 1 -- 172.21.15.146:0/3711981803 wait complete. 2024-04-25T15:23:26.246 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-04-25T15:23:26.033674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297880,"kb_used_data":6824,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754676840,"statfs":{"total":773094113280,"available":772789084160,"internally_reserved":0,"allocated":6987776,"data_stored":3810784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":297913906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.097635"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":186,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-25T15:23:16.243667+0000","last_change":"2024-04-25T15:19:35.053937+0000","last_active":"2024-04-25T15:23:16.243667+0000","last_peered":"2024-04-25T15:23:16.243667+0000","last_clean":"2024-04-25T15:23:16.243667+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:23:16.243667+0000","last_undegraded":"2024-04-25T15:23:16.243667+0000","last_fullsized":"2024-04-25T15:23:16.243667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:19:34.040957+0000","last_clean_scrub_stamp":"2024-04-25T15:19:34.040957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:28:26.864152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40212,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331628,"statfs":{"total":96636764160,"available":96595587072,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40208,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:23:26.248 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T15:19:34.040957+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=19, tm_sec=34, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=23, tm_sec=21, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:23:26.248 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T15:23:46.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:23:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.371+0000 7f478469a640 1 Processor -- start 2024-04-25T15:23:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.371+0000 7f478469a640 1 -- start start 2024-04-25T15:23:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f478469a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f477c0cf1d0 0x7f477c0b7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f478469a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c161b30 0x7f477c161f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f478469a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f477c0d1420 con 0x7f477c0d1cc0 2024-04-25T15:23:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f478469a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f477c0ced40 con 0x7f477c0cf1d0 2024-04-25T15:23:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f478469a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f477c0d1700 con 0x7f477c161b30 2024-04-25T15:23:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4782436640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f477c0cf1d0 0x7f477c0b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4781c35640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f477c0d1cc0 0x7f477c0d20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43244/0 (socket says 172.21.15.146:43244) 2024-04-25T15:23:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4781c35640 1 -- 172.21.15.146:0/2142622761 learned_addr learned my addr 172.21.15.146:0/2142622761 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-04-25T15:23:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4782c37640 1 --2- 172.21.15.146:0/2142622761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c161b30 0x7f477c161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4782436640 1 -- 172.21.15.146:0/2142622761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c161b30 msgr2=0x7f477c161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4782436640 1 --2- 172.21.15.146:0/2142622761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c161b30 0x7f477c161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4782436640 1 -- 172.21.15.146:0/2142622761 >> v1:172.21.15.146:6789/0 conn(0x7f477c0d1cc0 legacy=0x7f477c0d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4782436640 1 -- 172.21.15.146:0/2142622761 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c0d19e0 con 0x7f477c0cf1d0 2024-04-25T15:23:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.373+0000 7f4781434640 1 -- 172.21.15.146:0/2142622761 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508738625 0 0) 0x7f477c0d1420 con 0x7f477c0d1cc0 2024-04-25T15:23:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.374+0000 7f4782436640 1 --2- 172.21.15.146:0/2142622761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cf1d0 0x7f477c0b7350 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4778009d90 tx=0x7f477802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7dbbf0b384b37ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.374+0000 7f4781434640 1 -- 172.21.15.146:0/2142622761 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778007740 con 0x7f477c0cf1d0 2024-04-25T15:23:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.374+0000 7f4781434640 1 -- 172.21.15.146:0/2142622761 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47780078e0 con 0x7f477c0cf1d0 2024-04-25T15:23:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.374+0000 7f4781434640 1 -- 172.21.15.146:0/2142622761 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778007cb0 con 0x7f477c0cf1d0 2024-04-25T15:23:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 -- 172.21.15.146:0/2142622761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cf1d0 msgr2=0x7f477c0b7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 --2- 172.21.15.146:0/2142622761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cf1d0 0x7f477c0b7350 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4778009d90 tx=0x7f477802a660 comp rx=0 tx=0).stop 2024-04-25T15:23:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 -- 172.21.15.146:0/2142622761 shutdown_connections 2024-04-25T15:23:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 --2- 172.21.15.146:0/2142622761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c161b30 0x7f477c161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 --2- 172.21.15.146:0/2142622761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c0cf1d0 0x7f477c0b7350 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 -- 172.21.15.146:0/2142622761 >> 172.21.15.146:0/2142622761 conn(0x7f477c09b2b0 msgr2=0x7f477c0b6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 -- 172.21.15.146:0/2142622761 shutdown_connections 2024-04-25T15:23:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.375+0000 7f478469a640 1 -- 172.21.15.146:0/2142622761 wait complete. 2024-04-25T15:23:46.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.377+0000 7f478469a640 1 Processor -- start 2024-04-25T15:23:46.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.377+0000 7f478469a640 1 -- start start 2024-04-25T15:23:46.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f478469a640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c0cf1d0 0x7f477c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f478469a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f477c0d1cc0 0x7f477c1e8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f478469a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 0x7f477c14de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f478469a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f477c0cf5e0 con 0x7f477c161b30 2024-04-25T15:23:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f478469a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f477c0b7890 con 0x7f477c0d1cc0 2024-04-25T15:23:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f478469a640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7f477c15e310 con 0x7f477c0cf1d0 2024-04-25T15:23:46.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 0x7f477c14de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 0x7f477c14de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38006/0 (socket says 172.21.15.146:38006) 2024-04-25T15:23:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 -- 172.21.15.146:0/240010190 learned_addr learned my addr 172.21.15.146:0/240010190 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782436640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c0cf1d0 0x7f477c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4781c35640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f477c0d1cc0 0x7f477c1e8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 -- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c0cf1d0 msgr2=0x7f477c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c0cf1d0 0x7f477c0cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 -- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f477c0d1cc0 msgr2=0x7f477c1e8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f477c0d1cc0 0x7f477c1e8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4782c37640 1 -- 172.21.15.146:0/240010190 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c1e8f90 con 0x7f477c161b30 2024-04-25T15:23:46.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.378+0000 7f4781c35640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f477c0d1cc0 0x7f477c1e8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:23:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.379+0000 7f4782c37640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 0x7f477c14de00 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f477400d4b0 tx=0x7f477400d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.379+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740125f0 con 0x7f477c161b30 2024-04-25T15:23:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.379+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4774019070 con 0x7f477c161b30 2024-04-25T15:23:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.379+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f477c0ca7e0 con 0x7f477c161b30 2024-04-25T15:23:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.379+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740129d0 con 0x7f477c161b30 2024-04-25T15:23:46.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.379+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f477c0caac0 con 0x7f477c161b30 2024-04-25T15:23:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.380+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f477c06a0e0 con 0x7f477c161b30 2024-04-25T15:23:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.381+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7f4774012b70 con 0x7f477c161b30 2024-04-25T15:23:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.381+0000 7f4772ffd640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47600745d0 0x7f4760076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.381+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7f47740954f0 con 0x7f477c161b30 2024-04-25T15:23:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.381+0000 7f4782436640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47600745d0 0x7f4760076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.382+0000 7f4782436640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47600745d0 0x7f4760076ae0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4778034000 tx=0x7f477802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:46.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.386+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f477405fd20 con 0x7f477c161b30 2024-04-25T15:23:46.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:46.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:46.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.558+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 --> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f477c0d1200 con 0x7f47600745d0 2024-04-25T15:23:46.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.559+0000 7f4772ffd640 1 -- 172.21.15.146:0/240010190 <== mgr.4102 v2:172.21.15.146:6832/1999496207 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13344 (secure 0 0 0) 0x7f477c0d1200 con 0x7f47600745d0 2024-04-25T15:23:46.561 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:46.561 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-25T15:23:46.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47600745d0 msgr2=0x7f4760076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47600745d0 0x7f4760076ae0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4778034000 tx=0x7f477802f000 comp rx=0 tx=0).stop 2024-04-25T15:23:46.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 msgr2=0x7f477c14de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 0x7f477c14de00 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f477400d4b0 tx=0x7f477400d970 comp rx=0 tx=0).stop 2024-04-25T15:23:46.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 shutdown_connections 2024-04-25T15:23:46.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7f47600745d0 0x7f4760076ae0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477c161b30 0x7f477c14de00 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f477c0d1cc0 0x7f477c1e8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 --2- 172.21.15.146:0/240010190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f477c0cf1d0 0x7f477c0cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 >> 172.21.15.146:0/240010190 conn(0x7f477c09b2b0 msgr2=0x7f477c0b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 shutdown_connections 2024-04-25T15:23:46.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.561+0000 7f478469a640 1 -- 172.21.15.146:0/240010190 wait complete. 2024-04-25T15:23:46.574 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-04-25T15:23:46.037968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297868,"kb_used_data":6824,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754676852,"statfs":{"total":773094113280,"available":772789096448,"internally_reserved":0,"allocated":6987776,"data_stored":3810784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":297913906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002580"},"pg_stats":[{"pgid":"1.0","version":"60'74","reported_seq":198,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-25T15:23:26.383095+0000","last_change":"2024-04-25T15:23:26.383095+0000","last_active":"2024-04-25T15:23:26.383095+0000","last_peered":"2024-04-25T15:23:26.383095+0000","last_clean":"2024-04-25T15:23:26.383095+0000","last_became_active":"2024-04-25T15:19:35.052908+0000","last_became_peered":"2024-04-25T15:19:35.052908+0000","last_unstale":"2024-04-25T15:23:26.383095+0000","last_undegraded":"2024-04-25T15:23:26.383095+0000","last_fullsized":"2024-04-25T15:23:26.383095+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'74","last_scrub_stamp":"2024-04-25T15:23:26.382963+0000","last_deep_scrub":"60'74","last_deep_scrub_stamp":"2024-04-25T15:23:26.382963+0000","last_clean_scrub_stamp":"2024-04-25T15:23:26.382963+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:14.328229+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40212,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331628,"statfs":{"total":96636764160,"available":96595587072,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40208,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36052,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335788,"statfs":{"total":96636764160,"available":96599846912,"internally_reserved":0,"allocated":724992,"data_stored":328756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36624,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335216,"statfs":{"total":96636764160,"available":96599261184,"internally_reserved":0,"allocated":1318912,"data_stored":919124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:23:46.575 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T15:23:46.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.758+0000 7fbb59727640 1 Processor -- start 2024-04-25T15:23:46.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.758+0000 7fbb59727640 1 -- start start 2024-04-25T15:23:46.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb59727640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbb54163c00 0x7fbb5406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb59727640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb540bb450 0x7fbb540d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb59727640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5406dea0 con 0x7fbb540d0de0 2024-04-25T15:23:46.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb59727640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbb540b5230 con 0x7fbb540bb450 2024-04-25T15:23:46.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb59727640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5406c8c0 con 0x7fbb54163c00 2024-04-25T15:23:46.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb540bb450 0x7fbb540d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb540bb450 0x7fbb540d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38010/0 (socket says 172.21.15.146:38010) 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 -- 172.21.15.146:0/255978956 learned_addr learned my addr 172.21.15.146:0/255978956 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb537fe640 1 --1- 172.21.15.146:0/255978956 >> v1:172.21.15.146:6789/0 conn(0x7fbb540d0de0 0x7fbb540b8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43260/0 (socket says 172.21.15.146:43260) 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 -- 172.21.15.146:0/255978956 >> v2:172.21.15.177:3300/0 conn(0x7fbb54163c00 msgr2=0x7fbb5406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb52ffd640 1 --2- 172.21.15.146:0/255978956 >> v2:172.21.15.177:3300/0 conn(0x7fbb54163c00 0x7fbb5406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb51ffb640 1 -- 172.21.15.146:0/255978956 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2411656257 0 0) 0x7fbb5406dea0 con 0x7fbb540d0de0 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 --2- 172.21.15.146:0/255978956 >> v2:172.21.15.177:3300/0 conn(0x7fbb54163c00 0x7fbb5406d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 -- 172.21.15.146:0/255978956 >> v1:172.21.15.146:6789/0 conn(0x7fbb540d0de0 legacy=0x7fbb540b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb527fc640 1 -- 172.21.15.146:0/255978956 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5406cba0 con 0x7fbb540bb450 2024-04-25T15:23:46.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.760+0000 7fbb52ffd640 1 --2- 172.21.15.146:0/255978956 >> v2:172.21.15.177:3300/0 conn(0x7fbb54163c00 0x7fbb5406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:46.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.761+0000 7fbb527fc640 1 --2- 172.21.15.146:0/255978956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb540d08a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fbb5408e920 tx=0x7fbb4802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2779f902d3d8377 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:46.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.761+0000 7fbb51ffb640 1 -- 172.21.15.146:0/255978956 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48037070 con 0x7fbb540bb450 2024-04-25T15:23:46.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.761+0000 7fbb51ffb640 1 -- 172.21.15.146:0/255978956 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb48032440 con 0x7fbb540bb450 2024-04-25T15:23:46.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.761+0000 7fbb51ffb640 1 -- 172.21.15.146:0/255978956 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48032810 con 0x7fbb540bb450 2024-04-25T15:23:46.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 -- 172.21.15.146:0/255978956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 msgr2=0x7fbb540d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 --2- 172.21.15.146:0/255978956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb540d08a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fbb5408e920 tx=0x7fbb4802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 -- 172.21.15.146:0/255978956 shutdown_connections 2024-04-25T15:23:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 --2- 172.21.15.146:0/255978956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb540d08a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 --2- 172.21.15.146:0/255978956 >> v2:172.21.15.177:3300/0 conn(0x7fbb54163c00 0x7fbb5406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 -- 172.21.15.146:0/255978956 >> 172.21.15.146:0/255978956 conn(0x7fbb540b7970 msgr2=0x7fbb540b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 -- 172.21.15.146:0/255978956 shutdown_connections 2024-04-25T15:23:46.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.762+0000 7fbb59727640 1 -- 172.21.15.146:0/255978956 wait complete. 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.764+0000 7fbb59727640 1 Processor -- start 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.764+0000 7fbb59727640 1 -- start start 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb59727640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb541f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb59727640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb540d0de0 0x7fbb540ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb59727640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb54163c00 0x7fbb541f9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb59727640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5406c550 con 0x7fbb540bb450 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb59727640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5406d160 con 0x7fbb54163c00 2024-04-25T15:23:46.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb59727640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fbb540bb860 con 0x7fbb540d0de0 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb541f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb541f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38024/0 (socket says 172.21.15.146:38024) 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 -- 172.21.15.146:0/2029487422 learned_addr learned my addr 172.21.15.146:0/2029487422 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb537fe640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb54163c00 0x7fbb541f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb527fc640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb540d0de0 0x7fbb540ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 -- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb540d0de0 msgr2=0x7fbb540ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb540d0de0 0x7fbb540ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 -- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb54163c00 msgr2=0x7fbb541f9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb54163c00 0x7fbb541f9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb52ffd640 1 -- 172.21.15.146:0/2029487422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb54161aa0 con 0x7fbb540bb450 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.765+0000 7fbb527fc640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb540d0de0 0x7fbb540ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.766+0000 7fbb52ffd640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb541f3580 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400ce10 tx=0x7fbb4400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.766+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400f890 con 0x7fbb540bb450 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.766+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4400fa30 con 0x7fbb540bb450 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.766+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb54161d80 con 0x7fbb540bb450 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.766+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400fe20 con 0x7fbb540bb450 2024-04-25T15:23:46.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.766+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb54162340 con 0x7fbb540bb450 2024-04-25T15:23:46.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.767+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5406a0e0 con 0x7fbb540bb450 2024-04-25T15:23:46.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.767+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fbb44018400 con 0x7fbb540bb450 2024-04-25T15:23:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.768+0000 7fbb3b7fe640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb40074600 0x7fbb40076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.768+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fbb440960e0 con 0x7fbb540bb450 2024-04-25T15:23:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.768+0000 7fbb527fc640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb40074600 0x7fbb40076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.768+0000 7fbb527fc640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb40074600 0x7fbb40076b10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbb48034000 tx=0x7fbb4802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:46.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.773+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb44014030 con 0x7fbb540bb450 2024-04-25T15:23:46.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.948+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fbb540bad40 con 0x7fbb540bb450 2024-04-25T15:23:46.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.952+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v28)=0 v28) ==== 123+0+0 (secure 0 0 0) 0x7fbb4400ae20 con 0x7fbb540bb450 2024-04-25T15:23:46.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.953+0000 7fbb3b7fe640 1 -- 172.21.15.146:0/2029487422 <== mon.0 v2:172.21.15.146:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fbb4405fc10 con 0x7fbb540bb450 2024-04-25T15:23:46.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb40074600 msgr2=0x7fbb40076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb40074600 0x7fbb40076b10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbb48034000 tx=0x7fbb4802e000 comp rx=0 tx=0).stop 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 msgr2=0x7fbb541f3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb541f3580 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400ce10 tx=0x7fbb4400a650 comp rx=0 tx=0).stop 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 shutdown_connections 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fbb40074600 0x7fbb40076b10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb54163c00 0x7fbb541f9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb540d0de0 0x7fbb540ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 --2- 172.21.15.146:0/2029487422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb540bb450 0x7fbb541f3580 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:46.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.954+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 >> 172.21.15.146:0/2029487422 conn(0x7fbb540b7970 msgr2=0x7fbb540b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:46.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.955+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 shutdown_connections 2024-04-25T15:23:46.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:46.955+0000 7fbb59727640 1 -- 172.21.15.146:0/2029487422 wait complete. 2024-04-25T15:23:46.968 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:23:46.969 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:23:47.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.075+0000 7fb40991f640 1 Processor -- start 2024-04-25T15:23:47.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.075+0000 7fb40991f640 1 -- start start 2024-04-25T15:23:47.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb40991f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb4040d12a0 0x7fb40415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:47.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb40991f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 0x7fb40406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:47.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb40991f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4040d5000 con 0x7fb4040b45d0 2024-04-25T15:23:47.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb40991f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 -- 0x7fb4040d4db0 con 0x7fb4040d12a0 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb40991f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb4040d52e0 con 0x7fb4040d0e10 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb4037fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 0x7fb40406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb4040d12a0 0x7fb40415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb4027fc640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fb4040b45d0 0x7fb4040d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.146:60948/0 (socket says 172.21.15.146:60948) 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb4040d12a0 0x7fb40415ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39852/0 (socket says 172.21.15.146:39852) 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb4037fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 0x7fb40406cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.146:35142/0 (socket says 172.21.15.146:35142) 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 -- 172.21.15.146:0/3450789886 learned_addr learned my addr 172.21.15.146:0/3450789886 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:47.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 -- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 msgr2=0x7fb40406cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:23:47.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 --2- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 0x7fb40406cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 -- 172.21.15.146:0/3450789886 >> v1:172.21.15.177:6789/0 conn(0x7fb4040b45d0 legacy=0x7fb4040d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:47.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb402ffd640 1 -- 172.21.15.146:0/3450789886 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40406c5f0 con 0x7fb4040d12a0 2024-04-25T15:23:47.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.077+0000 7fb4037fe640 1 --2- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 0x7fb40406cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:47.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.078+0000 7fb402ffd640 1 --2- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040d12a0 0x7fb40415ff70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009870 tx=0x7fb3f402a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33e7dbbc267770e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.078+0000 7fb401ffb640 1 -- 172.21.15.146:0/3450789886 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4037070 con 0x7fb4040d12a0 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.078+0000 7fb401ffb640 1 -- 172.21.15.146:0/3450789886 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb3f4032440 con 0x7fb4040d12a0 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.078+0000 7fb401ffb640 1 -- 172.21.15.146:0/3450789886 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4032830 con 0x7fb4040d12a0 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 -- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040d12a0 msgr2=0x7fb40415ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 --2- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040d12a0 0x7fb40415ff70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009870 tx=0x7fb3f402a130 comp rx=0 tx=0).stop 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 -- 172.21.15.146:0/3450789886 shutdown_connections 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 --2- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb4040d0e10 0x7fb40406cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 --2- 172.21.15.146:0/3450789886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040d12a0 0x7fb40415ff70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 -- 172.21.15.146:0/3450789886 >> 172.21.15.146:0/3450789886 conn(0x7fb4040d63b0 msgr2=0x7fb4040d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:47.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 -- 172.21.15.146:0/3450789886 shutdown_connections 2024-04-25T15:23:47.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.079+0000 7fb40991f640 1 -- 172.21.15.146:0/3450789886 wait complete. 2024-04-25T15:23:47.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.081+0000 7fb40991f640 1 Processor -- start 2024-04-25T15:23:47.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.081+0000 7fb40991f640 1 -- start start 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb40991f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040b45d0 0x7fb40414c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb40991f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 0x7fb40414c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb40991f640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb40414f660 0x7fb40414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb40991f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb4040b9090 con 0x7fb4040d12a0 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb40991f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb4040b8f10 con 0x7fb4040b45d0 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb40991f640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 -- 0x7fb4040b4b20 con 0x7fb40414f660 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb4027fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 0x7fb40414c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:47.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb4027fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 0x7fb40414c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38038/0 (socket says 172.21.15.146:38038) 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb4037fe640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb40414f660 0x7fb40414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb4027fc640 1 -- 172.21.15.146:0/749804075 learned_addr learned my addr 172.21.15.146:0/749804075 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.082+0000 7fb402ffd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040b45d0 0x7fb40414c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4037fe640 1 -- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040b45d0 msgr2=0x7fb40414c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4037fe640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040b45d0 0x7fb40414c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4037fe640 1 -- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 msgr2=0x7fb40414c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4037fe640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 0x7fb40414c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4037fe640 1 -- 172.21.15.146:0/749804075 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4040cd620 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4027fc640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 0x7fb40414c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb4037fe640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb40414f660 0x7fb40414fa70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00ef90 tx=0x7fb3fc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc018030 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb3fc00cd30 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40414ffb0 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc003bd0 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.083+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40414a560 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.084+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4040cd7f0 con 0x7fb40414f660 2024-04-25T15:23:47.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.085+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 7) ==== 96884+0+0 (secure 0 0 0) 0x7fb3fc005d10 con 0x7fb40414f660 2024-04-25T15:23:47.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.085+0000 7fb3ebfff640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3f00745d0 0x7fb3f0076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:23:47.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.085+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 5421+0+0 (secure 0 0 0) 0x7fb3fc095bf0 con 0x7fb40414f660 2024-04-25T15:23:47.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.085+0000 7fb402ffd640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3f00745d0 0x7fb3f0076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:23:47.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.086+0000 7fb402ffd640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3f00745d0 0x7fb3f0076ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4034000 tx=0x7fb3f402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:23:47.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.090+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3fc022070 con 0x7fb40414f660 2024-04-25T15:23:47.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.301+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb40406a0e0 con 0x7fb40414f660 2024-04-25T15:23:47.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.302+0000 7fb3ebfff640 1 -- 172.21.15.146:0/749804075 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+3878 (secure 0 0 0) 0x7fb3fc010040 con 0x7fb40414f660 2024-04-25T15:23:47.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-25T15:23:47.304 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5441,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/4073326817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":4073326817},{"type":"v1","addr":"172.21.15.146:6837","nonce":4073326817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5444,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2063095308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2063095308},{"type":"v1","addr":"172.21.15.177:6834","nonce":2063095308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":5953,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/402650735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":402650735},{"type":"v1","addr":"172.21.15.177:6835","nonce":402650735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26},{"gid":7074,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2053993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2053993377},{"type":"v1","addr":"172.21.15.146:6836","nonce":2053993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[]} 2024-04-25T15:23:47.304 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 38 2024-04-25T15:23:47.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3f00745d0 msgr2=0x7fb3f0076ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:47.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3f00745d0 0x7fb3f0076ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4034000 tx=0x7fb3f402e000 comp rx=0 tx=0).stop 2024-04-25T15:23:47.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb40414f660 msgr2=0x7fb40414fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:23:47.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb40414f660 0x7fb40414fa70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00ef90 tx=0x7fb3fc00c490 comp rx=0 tx=0).stop 2024-04-25T15:23:47.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 shutdown_connections 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:6832/1999496207,v1:172.21.15.146:6833/1999496207] conn(0x7fb3f00745d0 0x7fb3f0076ae0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb40414f660 0x7fb40414fa70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4040d12a0 0x7fb40414c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 --2- 172.21.15.146:0/749804075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4040b45d0 0x7fb40414c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 >> 172.21.15.146:0/749804075 conn(0x7fb4040d63b0 msgr2=0x7fb4040bab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 shutdown_connections 2024-04-25T15:23:47.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-25T15:23:47.304+0000 7fb40991f640 1 -- 172.21.15.146:0/749804075 wait complete. 2024-04-25T15:23:47.317 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-25T15:23:47.318 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T15:23:47.318 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:23:47.318 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.328 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:23:47.328 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:23:47.328 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.381 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:23:47.382 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:23:47.382 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.394 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:23:47.395 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:23:47.395 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.446 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:23:47.446 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T15:23:47.447 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T15:23:47.447 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.481 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T15:23:47.482 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T15:23:47.482 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.550 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T15:23:47.551 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T15:23:47.551 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.619 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T15:23:47.619 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T15:23:47.619 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.685 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T15:23:47.685 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T15:23:47.685 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.727 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T15:23:47.727 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T15:23:47.728 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.808 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T15:23:47.809 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T15:23:47.809 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.851 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T15:23:47.851 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T15:23:47.851 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.891 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T15:23:47.891 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T15:23:47.891 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T15:23:47.891 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.919 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T15:23:47.919 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T15:23:47.919 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.956 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T15:23:47.956 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T15:23:47.956 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T15:23:47.956 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:47.971 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T15:23:47.971 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T15:23:47.971 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:48.009 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T15:23:48.009 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T15:23:48.009 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:23:48.060 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T15:23:48.060 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T15:23:48.060 DEBUG:teuthology.orchestra.run.smithi146:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | head -n 1 2024-04-25T15:23:48.090 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:23:48.090 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T15:23:48.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:23:48.318 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T15:23:48.385 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:23:48.385 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T15:23:48.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:23:48.450 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T15:23:48.516 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:23:48.517 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T15:23:48.720 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:23:48.720 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T15:23:48.791 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:23:48.791 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T15:23:48.858 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:23:48.858 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T15:23:48.924 INFO:tasks.ceph:Archiving mon data... 2024-04-25T15:23:48.926 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/data/mon.a.tgz 2024-04-25T15:23:48.926 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-04-25T15:23:48.942 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.RwHzOjK74B 2024-04-25T15:23:48.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RwHzOjK74B 2024-04-25T15:23:49.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.RwHzOjK74B 2024-04-25T15:23:49.168 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.RwHzOjK74B is 454KB 2024-04-25T15:23:49.266 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.RwHzOjK74B 2024-04-25T15:23:49.280 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/data/mon.b.tgz 2024-04-25T15:23:49.281 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-25T15:23:49.296 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.tKp6vmX4Cr 2024-04-25T15:23:49.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tKp6vmX4Cr 2024-04-25T15:23:49.469 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.tKp6vmX4Cr 2024-04-25T15:23:49.551 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.tKp6vmX4Cr is 607KB 2024-04-25T15:23:49.652 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.tKp6vmX4Cr 2024-04-25T15:23:49.666 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/data/mon.c.tgz 2024-04-25T15:23:49.667 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-04-25T15:23:49.721 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.gZNmR6piuv 2024-04-25T15:23:49.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.gZNmR6piuv 2024-04-25T15:23:49.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.gZNmR6piuv 2024-04-25T15:23:49.971 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.gZNmR6piuv is 607KB 2024-04-25T15:23:50.070 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.gZNmR6piuv 2024-04-25T15:23:50.084 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T15:23:50.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T15:23:50.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T15:23:50.171 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T15:23:50.201 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T15:23:50.202 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T15:23:50.205 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/remote/smithi146/crash 2024-04-25T15:23:50.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T15:23:50.237 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/remote/smithi177/crash 2024-04-25T15:23:50.238 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T15:23:50.264 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T15:23:50.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:50.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:50.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:50.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T15:23:50.411 INFO:tasks.ceph:Compressing logs... 2024-04-25T15:23:50.411 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T15:23:50.431 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T15:23:50.452 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51492.log 2024-04-25T15:23:50.452 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49814.log 2024-04-25T15:23:50.504 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34797.log 2024-04-25T15:23:50.504 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41331.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34797.log.gz 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.3.41331.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.3.41331.log.gz 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34717.log 2024-04-25T15:23:50.505 INFO:teuthology.orchestra.run.smithi177.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.40874.log 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34717.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34717.log.gz 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.1.40874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42562.log 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34506.log 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.1.42562.log: 90.0%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.1.40874.log.gz 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42086.log 2024-04-25T15:23:50.506 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34506.log.gz 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr:gzip 91.2% -- replaced with /var/log/ceph/ceph-client.1.42562.log.gz 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.1.42086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42921.log 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34757.log 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.1.42086.log.gz 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.3.42921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34546.log 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34757.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34757.log.gz 2024-04-25T15:23:50.507 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34546.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34546.log.gz 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.3.42921.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.1.43864.log 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34677.log 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.1.43864.log: 93.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44680.log 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34677.log: 89.0% -- replaced with /var/log/ceph/ceph-client.1.43864.log.gz 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34677.log.gz 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44340.log 2024-04-25T15:23:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34586.log 2024-04-25T15:23:50.509 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.3.44680.log: /var/log/ceph/ceph-client.1.44340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34637.log 2024-04-25T15:23:50.509 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34586.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34586.log.gz 2024-04-25T15:23:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T15:23:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34637.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34637.log.gz 2024-04-25T15:23:50.529 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T15:23:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43208.log 2024-04-25T15:23:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60281.log 2024-04-25T15:23:50.548 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51492.log.gz 2024-04-25T15:23:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-04-25T15:23:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57653.log 2024-04-25T15:23:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50862.log 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49814.log.gz 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60281.log: /var/log/ceph/ceph-client.admin.40939.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43208.log.gz 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60281.log.gz 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55109.log 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.0.57653.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60977.log 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50862.log.gz 2024-04-25T15:23:50.549 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55109.log.gz 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60977.log: 89.1% -- replaced with /var/log/ceph/ceph-client.0.57653.log.gz 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44254.log 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60977.log.gz 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63404.log 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59361.log 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44254.log.gz 2024-04-25T15:23:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48242.log 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63404.log.gz 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.0.59361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60692.log 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50522.log 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48242.log.gz 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57393.log 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60692.log.gz 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49940.log 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57393.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.56195.log 2024-04-25T15:23:50.551 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50522.log.gz 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49940.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57393.log.gz 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47307.log 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49940.log.gz 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47307.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56195.log.gz 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60013.log 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42047.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47307.log.gz 2024-04-25T15:23:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56069.log 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52543.log 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51030.log 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52543.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56069.log.gz 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60013.log.gz 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45835.log 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52543.log.gz 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51300.log 2024-04-25T15:23:50.553 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51030.log.gz 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.2.45835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.59786.log 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61433.log 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.2.59786.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51300.log.gz 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.2.45835.log.gz 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42257.log 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46949.log 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61433.log.gz 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-04-25T15:23:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz /var/log/ceph/ceph-client.admin.52501.log 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46949.log: /var/log/ceph/ceph-client.admin.44818.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46949.log.gz 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60572.log 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49016.log 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52501.log: /var/log/ceph/ceph-client.admin.60572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46319.log 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52501.log.gz 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60572.log.gz 2024-04-25T15:23:50.555 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49646.log 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49016.log.gz 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48686.log 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52375.log 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49646.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46319.log.gz 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48686.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49646.log.gz 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48686.log.gz 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2024-04-25T15:23:50.556 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52375.log: /var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56279.log 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52375.log.gz 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40725.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62758.log 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56279.log: /var/log/ceph/ceph-client.admin.44692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46151.log 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56279.log.gz 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62758.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-04-25T15:23:50.557 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59526.log 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62758.log.gz 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46151.log: -- /var/log/ceph/ceph-client.admin.54663.log 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41295.log 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46151.log.gz 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54663.log: gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.46865.log -- replaced with /var/log/ceph/ceph-client.admin.59526.log.gz 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54663.log.gz 2024-04-25T15:23:50.558 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46697.log 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46865.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41295.log.gz 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50269.log 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46865.log.gz 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55901.log 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46697.log.gz 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42994.log 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2%/var/log/ceph/ceph-client.admin.55901.log: -- replaced with /var/log/ceph/ceph-client.admin.50269.log.gz 2024-04-25T15:23:50.559 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55901.log.gz 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43543.log 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61391.log 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42994.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49394.log 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42994.log.gz 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43543.log.gz 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45957.log 2024-04-25T15:23:50.560 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61391.log: /var/log/ceph/ceph-client.admin.49394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46991.log 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61391.log.gz 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49394.log.gz 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46907.log 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45957.log.gz 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49604.log 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46991.log: /var/log/ceph/ceph-client.admin.46907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47210.log 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46991.log.gz 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46907.log.gz 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2024-04-25T15:23:50.561 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49604.log.gz 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47618.log 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47210.log: /var/log/ceph/ceph-client.admin.44398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57117.log 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47210.log.gz 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47618.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58942.log 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47618.log.gz 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T15:23:50.562 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57117.log: /var/log/ceph/ceph-client.admin.58942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61643.log 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57117.log.gz 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58942.log.gz 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54983.log 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58690.log 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54983.log: gzip -5 89.6% -- replaced with /var/log/ceph/ceph-client.admin.61643.log.gz 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose 85.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr: -- /var/log/ceph/ceph-client.admin.57159.log 2024-04-25T15:23:50.563 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54983.log.gz 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57159.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58690.log.gz 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49982.log 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57159.log.gz/var/log/ceph/ceph-client.admin.41253.log: 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50988.log 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49982.log: /var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49268.log 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49982.log.gz 2024-04-25T15:23:50.564 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61601.log 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50988.log.gz -5 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52789.log 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49268.log: /var/log/ceph/ceph-client.admin.61601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47033.log 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49268.log.gz 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52789.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61601.log.gz 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58648.log 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47033.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52789.log.gz 2024-04-25T15:23:50.565 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45337.log 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47033.log.gz 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52275.log 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58648.log.gz 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58118.log 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.0.45337.log: /var/log/ceph/ceph-client.admin.52275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48400.log 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52275.log.gz 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58118.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.45337.log.gz 2024-04-25T15:23:50.566 INFO:teuthology.orchestra.run.smithi146.stderr:gzip/var/log/ceph/ceph-client.admin.48400.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48586.log 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58118.log.gz 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.2.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48400.log.gz 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52873.log 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5/var/log/ceph/ceph-client.admin.48586.log: --verbose -- /var/log/ceph/ceph-client.admin.58034.log 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61475.log 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48586.log.gz 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58034.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52873.log.gz 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60488.log 2024-04-25T15:23:50.567 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58034.log.gz 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46571.log 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55817.log/var/log/ceph/ceph-client.admin.60488.log: 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.61475.log.gz 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54290.log 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46571.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60488.log.gz 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49478.log 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46571.log.gz 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55817.log.gz 2024-04-25T15:23:50.568 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48974.log 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46739.log 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49478.log: /var/log/ceph/ceph-client.admin.48974.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54290.log.gz 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49478.log.gz 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.54909.log 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48974.log.gz 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42131.log 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46739.log.gz 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44566.log 2024-04-25T15:23:50.569 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54909.log.gz 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43554.log 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61914.log 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44566.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42131.log.gz 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41337.log 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44566.log.gz 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61914.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43554.log.gz 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61914.log.gz 2024-04-25T15:23:50.570 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54054.log 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41337.log.gz 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57894.log 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57894.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60734.log 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54054.log.gz 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44776.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57894.log.gz 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50648.log 2024-04-25T15:23:50.571 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60734.log: /var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47744.log 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44440.log 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60734.log.gz 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47744.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50648.log.gz 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47160.log 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47744.log.gz 2024-04-25T15:23:50.572 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53988.log 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47160.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44440.log.gz 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52035.log 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47160.log.gz 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55985.log 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61019.log 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0%/var/log/ceph/ceph-client.admin.55985.log: -- replaced with /var/log/ceph/ceph-client.admin.52035.log.gz 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53988.log.gz 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58858.log 2024-04-25T15:23:50.573 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55985.log.gz 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52191.log 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58858.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61019.log.gz 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55733.log 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52191.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58858.log.gz 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59484.log 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52191.log.gz 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54867.log 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50066.log 2024-04-25T15:23:50.574 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59484.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55733.log.gz 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51198.log 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59484.log.gz 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50066.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54867.log.gz 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62926.log 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51198.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50066.log.gz 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50690.log 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62926.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.51198.log.gz 2024-04-25T15:23:50.575 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52149.log 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62926.log.gz 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49772.log 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52149.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50690.log.gz 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42399.log 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52149.log.gz 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46445.log 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49772.log.gz 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:50.576 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56662.log 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42399.log.gz 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57033.log 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56662.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46445.log.gz 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56662.log.gz 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56903.log 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40517.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57033.log.gz 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57285.log 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-04-25T15:23:50.577 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43553.log 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56903.log: /var/log/ceph/ceph-client.admin.57285.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56903.log.gz 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.56764.log 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8%/var/log/ceph/ceph-client.admin.43553.log: -- replaced with /var/log/ceph/ceph-client.admin.57285.log.gz 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42089.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43553.log.gz 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55859.log 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56764.log.gz 2024-04-25T15:23:50.578 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42089.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55859.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42089.log.gz 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49352.log 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55859.log.gz 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48484.log 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42604.log 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49352.log.gz 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61811.log 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48484.log: /var/log/ceph/ceph-client.admin.42604.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48484.log.gz 2024-04-25T15:23:50.579 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49520.log 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4%/var/log/ceph/ceph-client.admin.61811.log: -- replaced with /var/log/ceph/ceph-client.admin.42604.log.gz 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49100.log 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61811.log.gz 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55775.log 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49520.log.gz 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42525.log 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55775.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49100.log.gz 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50606.log 2024-04-25T15:23:50.580 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55775.log.gz 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42215.log 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50606.log: gzip -5 --verbose -- 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42525.log.gz 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.54376.log 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42215.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50606.log.gz 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50946.log 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42215.log.gz 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46823.log 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58402.log 2024-04-25T15:23:50.581 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4%/var/log/ceph/ceph-client.admin.46823.log: -- replaced with /var/log/ceph/ceph-client.admin.50946.log.gz 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49856.log 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54376.log.gz 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46823.log.gz 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57992.log 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58402.log: /var/log/ceph/ceph-client.admin.49856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52417.log 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49856.log.gz 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57992.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58402.log.gz 2024-04-25T15:23:50.582 INFO:teuthology.orchestra.run.smithi146.stderr: -- /var/log/ceph/ceph-client.admin.60195.log 87.5% 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57992.log.gz 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47434.log 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52417.log.gz 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61998.log 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47434.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60195.log.gz 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50734.log 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47434.log.gz 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-04-25T15:23:50.583 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61998.log.gz 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50734.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50734.log.gz 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.41860.log 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46109.log 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55691.log 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41860.log: gzip -5/var/log/ceph/ceph-client.admin.46109.log: --verbose -- /var/log/ceph/ceph-client.admin.43882.log 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55691.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.46109.log.gz 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48896.log 2024-04-25T15:23:50.584 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55691.log.gz 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43882.log: /var/log/ceph/ceph-client.admin.48896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48284.log 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.c.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.41860.log.gz 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43882.log.gz 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48896.log.gz 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47660.log 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60121.log 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47660.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48284.log.gz 2024-04-25T15:23:50.585 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47660.log.gz/var/log/ceph/ceph-client.admin.60121.log: 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51752.log 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60121.log.gz 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43887.log 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40329.log 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.0.51752.log: /var/log/ceph/ceph-client.admin.43887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42868.log 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40329.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43887.log.gz 2024-04-25T15:23:50.586 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40329.log.gz 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.0.51752.log.gz 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43770.log 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42357.log 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42868.log.gz 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43770.log.gz 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42357.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42357.log.gz/var/log/ceph/ceph-client.admin.48200.log: 2024-04-25T15:23:50.587 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48442.log 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52621.log 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48442.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48442.log.gz 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54783.log 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52621.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52621.log.gz 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55439.log 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54783.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54783.log.gz 2024-04-25T15:23:50.588 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42173.log 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55439.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55439.log.gz 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44524.log 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42173.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42173.log.gz 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47075.log 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44524.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44524.log.gz 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46235.log 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47075.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47075.log.gz 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62226.log 2024-04-25T15:23:50.589 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46235.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46235.log.gz 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42952.log 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62226.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62226.log.gz 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42952.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42952.log.gz 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58218.log 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40640.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61061.log 2024-04-25T15:23:50.590 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58218.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58218.log.gz 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47476.log 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61061.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61061.log.gz 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48770.log 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47476.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47476.log.gz 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48770.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48770.log.gz 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54825.log 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62800.log 2024-04-25T15:23:50.591 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54825.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54825.log.gz 2024-04-25T15:23:50.593 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43204.log 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62800.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62800.log.gz 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55943.log 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43204.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43204.log.gz 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63362.log 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55943.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55943.log.gz 2024-04-25T15:23:50.594 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47350.log 2024-04-25T15:23:50.595 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63362.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63362.log.gz 2024-04-25T15:23:50.595 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-04-25T15:23:50.595 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47350.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47350.log.gz 2024-04-25T15:23:50.607 INFO:teuthology.orchestra.run.smithi146.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T15:23:50.608 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55229.log 2024-04-25T15:23:50.608 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40598.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-04-25T15:23:50.611 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42826.log 2024-04-25T15:23:50.611 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55229.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55229.log.gz 2024-04-25T15:23:50.612 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58260.log 2024-04-25T15:23:50.612 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42826.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42826.log.gz 2024-04-25T15:23:50.612 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56578.log 2024-04-25T15:23:50.614 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58260.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58260.log.gz 2024-04-25T15:23:50.614 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54334.log 2024-04-25T15:23:50.614 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56578.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56578.log.gz 2024-04-25T15:23:50.614 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61517.log 2024-04-25T15:23:50.616 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54334.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54334.log.gz 2024-04-25T15:23:50.616 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47392.log 2024-04-25T15:23:50.617 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61517.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61517.log.gz 2024-04-25T15:23:50.617 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48326.log 2024-04-25T15:23:50.618 INFO:teuthology.orchestra.run.smithi177.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T15:23:50.621 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47392.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47392.log.gz 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43888.log 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48326.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48326.log.gz 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50480.log 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43888.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43888.log.gz 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54537.log 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50480.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50480.log.gz 2024-04-25T15:23:50.622 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44086.log 2024-04-25T15:23:50.625 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54537.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54537.log.gz 2024-04-25T15:23:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42647.log 2024-04-25T15:23:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44086.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44086.log.gz 2024-04-25T15:23:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63094.log 2024-04-25T15:23:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42647.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42647.log.gz 2024-04-25T15:23:50.626 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63052.log 2024-04-25T15:23:50.629 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63094.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63094.log.gz 2024-04-25T15:23:50.629 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53511.log 2024-04-25T15:23:50.629 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63052.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63052.log.gz 2024-04-25T15:23:50.630 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56153.log 2024-04-25T15:23:50.630 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53511.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53511.log.gz 2024-04-25T15:23:50.630 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50353.log 2024-04-25T15:23:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56153.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56153.log.gz 2024-04-25T15:23:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62142.log 2024-04-25T15:23:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50353.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50353.log.gz 2024-04-25T15:23:50.633 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50438.log 2024-04-25T15:23:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62142.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62142.log.gz 2024-04-25T15:23:50.634 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42691.log 2024-04-25T15:23:50.635 INFO:teuthology.orchestra.run.smithi177.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T15:23:50.639 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50438.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50438.log.gz 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42691.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42691.log.gz 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46529.log 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44128.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52233.log 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46529.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46529.log.gz 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52747.log 2024-04-25T15:23:50.640 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52233.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52233.log.gz 2024-04-25T15:23:50.641 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42741.log 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52747.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52747.log.gz 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63262.log 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42741.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42741.log.gz 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47948.log 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63262.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63262.log.gz 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62040.log 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47948.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47948.log.gz 2024-04-25T15:23:50.645 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61853.log 2024-04-25T15:23:50.648 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62040.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62040.log.gz 2024-04-25T15:23:50.649 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57201.log 2024-04-25T15:23:50.649 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61853.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61853.log.gz 2024-04-25T15:23:50.649 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55271.log 2024-04-25T15:23:50.649 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57201.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57201.log.gz 2024-04-25T15:23:50.649 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42441.log 2024-04-25T15:23:50.651 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55271.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55271.log.gz 2024-04-25T15:23:50.651 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55397.log 2024-04-25T15:23:50.651 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42441.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42441.log.gz 2024-04-25T15:23:50.651 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59101.log 2024-04-25T15:23:50.653 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55397.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55397.log.gz 2024-04-25T15:23:50.654 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43216.log 2024-04-25T15:23:50.654 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59101.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59101.log.gz 2024-04-25T15:23:50.654 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56987.log 2024-04-25T15:23:50.656 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43216.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43216.log.gz 2024-04-25T15:23:50.656 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49226.log 2024-04-25T15:23:50.656 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56987.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56987.log.gz 2024-04-25T15:23:50.656 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48854.log 2024-04-25T15:23:50.658 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49226.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49226.log.gz 2024-04-25T15:23:50.658 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53074.log 2024-04-25T15:23:50.658 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48854.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48854.log.gz 2024-04-25T15:23:50.660 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56945.log 2024-04-25T15:23:50.660 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53074.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53074.log.gz 2024-04-25T15:23:50.660 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T15:23:50.661 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56945.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56945.log.gz 2024-04-25T15:23:50.662 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43545.log 2024-04-25T15:23:50.675 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53468.log 2024-04-25T15:23:50.675 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43545.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43545.log.gz 2024-04-25T15:23:50.688 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62632.log 2024-04-25T15:23:50.689 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53468.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53468.log.gz 2024-04-25T15:23:50.690 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50024.log 2024-04-25T15:23:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62632.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49310.log 2024-04-25T15:23:50.702 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62632.log.gz 2024-04-25T15:23:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50024.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50024.log.gz 2024-04-25T15:23:50.709 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55151.log 2024-04-25T15:23:50.709 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49310.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49310.log.gz 2024-04-25T15:23:50.718 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60079.log 2024-04-25T15:23:50.718 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55151.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55151.log.gz 2024-04-25T15:23:50.727 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62268.log 2024-04-25T15:23:50.727 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60079.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60079.log.gz 2024-04-25T15:23:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58076.log 2024-04-25T15:23:50.736 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62268.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62268.log.gz 2024-04-25T15:23:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-04-25T15:23:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58076.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58076.log.gz 2024-04-25T15:23:50.764 INFO:teuthology.orchestra.run.smithi177.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T15:23:50.765 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56852.log 2024-04-25T15:23:50.765 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48032.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-04-25T15:23:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43548.log 2024-04-25T15:23:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56852.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56852.log.gz 2024-04-25T15:23:50.790 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43212.log 2024-04-25T15:23:50.790 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43548.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43548.log.gz 2024-04-25T15:23:50.808 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63220.log 2024-04-25T15:23:50.808 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43212.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43212.log.gz 2024-04-25T15:23:50.817 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log 2024-04-25T15:23:50.817 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63220.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63220.log.gz 2024-04-25T15:23:50.833 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50108.log 2024-04-25T15:23:50.833 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41041.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2024-04-25T15:23:50.842 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47864.log 2024-04-25T15:23:50.842 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50108.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50108.log.gz 2024-04-25T15:23:50.851 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58732.log 2024-04-25T15:23:50.851 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47864.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47864.log.gz 2024-04-25T15:23:50.861 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44734.log 2024-04-25T15:23:50.861 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58732.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58732.log.gz 2024-04-25T15:23:50.875 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2024-04-25T15:23:50.875 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44734.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44734.log.gz 2024-04-25T15:23:50.889 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58900.log 2024-04-25T15:23:50.889 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51993.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz 2024-04-25T15:23:50.899 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58176.log 2024-04-25T15:23:50.899 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58900.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58900.log.gz 2024-04-25T15:23:50.908 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51909.log 2024-04-25T15:23:50.908 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58176.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58176.log.gz 2024-04-25T15:23:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34667.log 2024-04-25T15:23:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51909.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51909.log.gz 2024-04-25T15:23:50.932 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55523.log 2024-04-25T15:23:50.932 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34667.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34667.log.gz 2024-04-25T15:23:50.951 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40421.log 2024-04-25T15:23:50.951 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55523.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55523.log.gz 2024-04-25T15:23:50.959 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49730.log 2024-04-25T15:23:50.972 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40421.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40421.log.gz 2024-04-25T15:23:50.973 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62464.log 2024-04-25T15:23:50.974 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49730.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49730.log.gz 2024-04-25T15:23:50.982 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58486.log 2024-04-25T15:23:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62464.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45459.log 2024-04-25T15:23:50.992 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58486.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58486.log.gz 2024-04-25T15:23:50.996 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62464.log.gz 2024-04-25T15:23:51.008 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62716.log 2024-04-25T15:23:51.009 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45459.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45459.log.gz 2024-04-25T15:23:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56416.log 2024-04-25T15:23:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62716.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62716.log.gz 2024-04-25T15:23:51.041 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54204.log 2024-04-25T15:23:51.042 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56416.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56416.log.gz 2024-04-25T15:23:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51114.log 2024-04-25T15:23:51.062 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54204.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52663.log 2024-04-25T15:23:51.062 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51114.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51114.log.gz 2024-04-25T15:23:51.064 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54204.log.gz 2024-04-25T15:23:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60325.log 2024-04-25T15:23:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-04-25T15:23:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60325.log: 91.1% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52663.log.gz 2024-04-25T15:23:51.089 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58344.log 2024-04-25T15:23:51.090 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40809.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-04-25T15:23:51.093 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60325.log.gz 2024-04-25T15:23:51.098 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62842.log 2024-04-25T15:23:51.105 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58344.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58344.log.gz 2024-04-25T15:23:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41211.log 2024-04-25T15:23:51.117 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62842.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53893.log 2024-04-25T15:23:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41211.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62842.log.gz 2024-04-25T15:23:51.130 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T15:23:51.130 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41211.log.gz 2024-04-25T15:23:51.132 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52705.log 2024-04-25T15:23:51.132 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53893.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53893.log.gz 2024-04-25T15:23:51.132 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57075.log 2024-04-25T15:23:51.133 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52705.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52705.log.gz 2024-04-25T15:23:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60237.log 2024-04-25T15:23:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57075.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57075.log.gz 2024-04-25T15:23:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51384.log 2024-04-25T15:23:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60237.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60237.log.gz 2024-04-25T15:23:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34770.log 2024-04-25T15:23:51.138 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51384.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51384.log.gz 2024-04-25T15:23:51.138 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46613.log 2024-04-25T15:23:51.138 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34770.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34770.log.gz 2024-04-25T15:23:51.138 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61103.log 2024-04-25T15:23:51.140 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46613.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46613.log.gz 2024-04-25T15:23:51.141 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55025.log 2024-04-25T15:23:51.141 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61103.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61103.log.gz 2024-04-25T15:23:51.141 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-04-25T15:23:51.141 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55025.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55025.log.gz 2024-04-25T15:23:51.141 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55067.log 2024-04-25T15:23:51.144 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44608.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-04-25T15:23:51.144 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-04-25T15:23:51.145 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55067.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55067.log.gz 2024-04-25T15:23:51.145 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46781.log 2024-04-25T15:23:51.145 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41169.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-04-25T15:23:51.145 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55607.log 2024-04-25T15:23:51.148 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46781.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46781.log.gz 2024-04-25T15:23:51.148 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46193.log 2024-04-25T15:23:51.148 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55607.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55607.log.gz 2024-04-25T15:23:51.148 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60446.log 2024-04-25T15:23:51.148 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46193.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46193.log.gz 2024-04-25T15:23:51.149 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62184.log 2024-04-25T15:23:51.151 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60446.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60446.log.gz 2024-04-25T15:23:51.151 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47518.log 2024-04-25T15:23:51.151 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62184.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62184.log.gz 2024-04-25T15:23:51.151 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40895.log 2024-04-25T15:23:51.152 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47518.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47518.log.gz 2024-04-25T15:23:51.153 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48544.log 2024-04-25T15:23:51.153 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40895.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40895.log.gz 2024-04-25T15:23:51.155 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51072.log 2024-04-25T15:23:51.155 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48544.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48544.log.gz 2024-04-25T15:23:51.155 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50396.log 2024-04-25T15:23:51.157 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51072.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51072.log.gz 2024-04-25T15:23:51.157 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2024-04-25T15:23:51.157 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50396.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50396.log.gz 2024-04-25T15:23:51.157 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60935.log 2024-04-25T15:23:51.160 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52459.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2024-04-25T15:23:51.160 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41127.log 2024-04-25T15:23:51.160 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60935.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60935.log.gz 2024-04-25T15:23:51.161 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49898.log 2024-04-25T15:23:51.161 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41127.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41127.log.gz 2024-04-25T15:23:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63178.log 2024-04-25T15:23:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49898.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49898.log.gz 2024-04-25T15:23:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52957.log 2024-04-25T15:23:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63178.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63178.log.gz 2024-04-25T15:23:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46403.log 2024-04-25T15:23:51.166 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52957.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52957.log.gz 2024-04-25T15:23:51.166 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2024-04-25T15:23:51.169 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46403.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46403.log.gz 2024-04-25T15:23:51.169 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49184.log 2024-04-25T15:23:51.169 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47576.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz 2024-04-25T15:23:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-04-25T15:23:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49184.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49184.log.gz 2024-04-25T15:23:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-04-25T15:23:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42910.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-04-25T15:23:51.172 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50904.log 2024-04-25T15:23:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47990.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-04-25T15:23:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57852.log 2024-04-25T15:23:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50904.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50904.log.gz 2024-04-25T15:23:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46361.log 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57852.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57852.log.gz 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50188.log 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46361.log: 92.9% -- replaced with /var/log/ceph/ceph-client.0.59361.log.gz 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46361.log.gz 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57327.log 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54248.log 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr: 89.7%/var/log/ceph/ceph-client.admin.57327.log: -- replaced with /var/log/ceph/ceph-client.admin.50188.log.gz 2024-04-25T15:23:51.179 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53345.log 2024-04-25T15:23:51.180 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54248.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.57327.log.gz 2024-04-25T15:23:51.180 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54248.log.gz 2024-04-25T15:23:51.180 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56620.log 2024-04-25T15:23:51.180 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.53345.log: -- /var/log/ceph/ceph-client.admin.43549.log 2024-04-25T15:23:51.180 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56620.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56620.log.gz 2024-04-25T15:23:51.180 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51426.log 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43549.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43549.log.gz 91.8% -- replaced with /var/log/ceph/ceph-client.0.53345.log.gz 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57936.log 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44338.log 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57936.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51426.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57936.log.gz 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.53771.log 2024-04-25T15:23:51.183 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54741.log 2024-04-25T15:23:51.184 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44338.log: /var/log/ceph/ceph-client.2.53771.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44338.log.gz 2024-04-25T15:23:51.184 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43215.log 2024-04-25T15:23:51.189 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54741.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54741.log.gz 2024-04-25T15:23:51.189 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2024-04-25T15:23:51.189 INFO:teuthology.orchestra.run.smithi146.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.2.53771.log.gz 2024-04-25T15:23:51.189 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62674.log 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40767.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43215.log.gz 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56027.log 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61727.log 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62674.log: /var/log/ceph/ceph-client.admin.56027.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62674.log.gz 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56027.log.gz 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59908.log 2024-04-25T15:23:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49142.log 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59908.log: /var/log/ceph/ceph-client.admin.61727.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61727.log.gz 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59908.log.gz 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49562.log 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55355.log 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49142.log: /var/log/ceph/ceph-client.admin.49562.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49142.log.gz 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49562.log.gz 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62968.log 2024-04-25T15:23:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42784.log 2024-04-25T15:23:51.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55355.log: /var/log/ceph/ceph-client.admin.62968.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55355.log.gz 2024-04-25T15:23:51.199 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62968.log.gz 2024-04-25T15:23:51.199 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46487.log 2024-04-25T15:23:51.199 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62590.log 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42784.log: /var/log/ceph/ceph-client.admin.46487.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42784.log.gz 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46487.log.gz 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43884.log 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58528.log 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43884.log: /var/log/ceph/ceph-client.admin.62590.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62590.log.gz 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43884.log.gz 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40851.log 2024-04-25T15:23:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55565.log 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58528.log: /var/log/ceph/ceph-client.admin.40851.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58528.log.gz 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40851.log.gz 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52915.log 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44945.log 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55565.log: /var/log/ceph/ceph-client.admin.52915.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55565.log.gz 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52915.log.gz 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57243.log 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52091.log 2024-04-25T15:23:51.201 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57243.log: /var/log/ceph/ceph-client.admin.44945.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44945.log.gz 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57243.log.gz 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54579.log 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43217.log 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52091.log: /var/log/ceph/ceph-client.admin.54579.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52091.log.gz 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54579.log.gz 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58570.log 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2024-04-25T15:23:51.202 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58570.log: /var/log/ceph/ceph-client.admin.43217.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43217.log.gz 2024-04-25T15:23:51.203 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58570.log.gz 2024-04-25T15:23:51.203 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61181.log 2024-04-25T15:23:51.203 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61956.log 2024-04-25T15:23:51.220 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43218.log: /var/log/ceph/ceph-client.admin.61181.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61181.log.gz 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46277.log 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61956.log: /var/log/ceph/ceph-client.admin.51951.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61956.log.gz 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54495.log 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55313.log 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54495.log: /var/log/ceph/ceph-client.admin.46277.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46277.log.gz 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54495.log.gz 2024-04-25T15:23:51.221 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54162.log 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58302.log 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55313.log: /var/log/ceph/ceph-client.admin.54162.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55313.log.gz 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54162.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62884.log 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55649.log 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58302.log: /var/log/ceph/ceph-client.admin.62884.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58302.log.gz 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58444.log 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62884.log.gz 2024-04-25T15:23:51.222 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61223.log 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58444.log: 88.3% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55649.log.gz 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58444.log.gz 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54621.log 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56237.log 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61223.log: /var/log/ceph/ceph-client.admin.54621.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61223.log.gz 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54621.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61559.log 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63010.log 2024-04-25T15:23:51.223 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61559.log: /var/log/ceph/ceph-client.admin.56237.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56237.log.gz 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61559.log.gz 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43890.log 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51156.log 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63010.log: /var/log/ceph/ceph-client.admin.43890.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63010.log.gz 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43890.log.gz 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60861.log 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58984.log 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51156.log: /var/log/ceph/ceph-client.admin.60861.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51156.log.gz 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61685.log 2024-04-25T15:23:51.224 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60861.log.gz 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60776.log 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58984.log: /var/log/ceph/ceph-client.admin.61685.log: 88.4% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61685.log.gz 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58984.log.gz 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34403.log 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56496.log 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60776.log.gz 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48812.log 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56496.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34403.log.gz 2024-04-25T15:23:51.225 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56496.log.gz 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62422.log 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28373.log 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48812.log: /var/log/ceph/ceph-client.admin.62422.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48812.log.gz 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62422.log.gz 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52831.log 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57810.log 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52831.log: /var/log/ceph/ceph.tmp-client.admin.28373.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28373.log.gz 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52831.log.gz 2024-04-25T15:23:51.226 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48158.log 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52317.log 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57810.log.gz 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40683.log 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52317.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48158.log.gz 88.4% 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52317.log.gz 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56808.log 2024-04-25T15:23:51.227 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40683.log.gz 2024-04-25T15:23:51.228 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43036.log 2024-04-25T15:23:51.228 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56808.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56808.log.gz 2024-04-25T15:23:51.256 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 2024-04-25T15:23:51.256 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43036.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43036.log.gz 2024-04-25T15:23:51.256 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50778.log 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41085.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40999.log 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50778.log.gz 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63136.log 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48628.log 2024-04-25T15:23:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63136.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40999.log.gz 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63136.log.gz 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50311.log 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43769.log 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48628.log: /var/log/ceph/ceph-client.admin.50311.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48628.log.gz 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50311.log.gz 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50820.log 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59026.log 2024-04-25T15:23:51.258 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43769.log: /var/log/ceph/ceph-client.admin.50820.log: 88.6% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50820.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43769.log.gz 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48116.log 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.59026.log: /var/log/ceph/ceph-client.admin.63304.log 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59026.log.gz 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51342.log 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63304.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48116.log.gz 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63304.log.gz 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62100.log 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47786.log 2024-04-25T15:23:51.259 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51342.log.gz 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47117.log 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47786.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62100.log.gz 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47786.log.gz 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60530.log 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41379.log 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47117.log: /var/log/ceph/ceph-client.admin.60530.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47117.log.gz 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60530.log.gz 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43550.log 2024-04-25T15:23:51.260 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54096.log 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43550.log: /var/log/ceph/ceph-client.admin.41379.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43550.log.gz 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56321.log 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54096.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41379.log.gz 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50564.log 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56321.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54096.log.gz 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56321.log.gz 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56111.log 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43214.log 2024-04-25T15:23:51.261 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50564.log.gz 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47906.log 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43214.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56111.log.gz 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43214.log.gz 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42483.log 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60818.log 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47906.log.gz 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61307.log 2024-04-25T15:23:51.262 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42483.log.gz 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56722.log 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61307.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60818.log.gz 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52999.log 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61307.log.gz 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49436.log 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52999.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56722.log.gz 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58774.log 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52999.log.gz 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60614.log 2024-04-25T15:23:51.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58774.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49436.log.gz 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58774.log.gz 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62506.log 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46655.log 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60614.log: /var/log/ceph/ceph-client.admin.62506.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60614.log.gz 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61769.log 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62506.log.gz 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51240.log 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61769.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46655.log.gz 2024-04-25T15:23:51.264 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61769.log.gz 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49688.log 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55481.log 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49688.log: /var/log/ceph/ceph-client.admin.51240.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51240.log.gz 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49688.log.gz 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61349.log 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44296.log 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55481.log: /var/log/ceph/ceph-client.admin.61349.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55481.log.gz 2024-04-25T15:23:51.265 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45999.log 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61349.log.gz 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42315.log 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45999.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44296.log.gz 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45999.log.gz 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62548.log 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49058.log 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42315.log: /var/log/ceph/ceph-client.admin.62548.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42315.log.gz 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62548.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43889.log 2024-04-25T15:23:51.266 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48728.log 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43889.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49058.log.gz 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43889.log.gz 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58816.log 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61265.log 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48728.log: /var/log/ceph/ceph-client.admin.58816.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48728.log.gz 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.58816.log.gz 2024-04-25T15:23:51.267 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61265.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61265.log.gz 2024-04-25T15:23:51.268 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.1.44340.log.gz 2024-04-25T15:23:51.961 INFO:teuthology.orchestra.run.smithi146.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.2.59786.log.gz 2024-04-25T15:23:52.144 INFO:teuthology.orchestra.run.smithi177.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T15:23:52.251 INFO:teuthology.orchestra.run.smithi177.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T15:23:52.324 INFO:teuthology.orchestra.run.smithi146.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T15:23:52.377 INFO:teuthology.orchestra.run.smithi177.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.3.44680.log.gz 2024-04-25T15:23:52.423 INFO:teuthology.orchestra.run.smithi177.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T15:23:52.438 INFO:teuthology.orchestra.run.smithi146.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T15:23:52.486 INFO:teuthology.orchestra.run.smithi177.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T15:23:52.488 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-25T15:23:52.488 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m2.023s 2024-04-25T15:23:52.488 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m10.215s 2024-04-25T15:23:52.488 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.315s 2024-04-25T15:23:52.819 INFO:teuthology.orchestra.run.smithi146.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T15:23:52.844 INFO:teuthology.orchestra.run.smithi146.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T15:23:59.891 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T15:23:59.892 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-25T15:23:59.893 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m9.450s 2024-04-25T15:23:59.893 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m20.009s 2024-04-25T15:23:59.893 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.943s 2024-04-25T15:23:59.893 INFO:tasks.ceph:Archiving logs... 2024-04-25T15:23:59.894 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/remote/smithi146/log 2024-04-25T15:23:59.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T15:24:03.010 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/remote/smithi177/log 2024-04-25T15:24:03.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T15:24:04.370 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T15:24:04.381 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T15:24:04.381 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T15:24:04.384 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T15:24:04.469 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T15:24:04.479 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T15:24:04.479 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:24:04.482 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:24:04.496 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:24:04.498 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:24:04.500 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:24:04.501 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-04-25T15:24:04.501 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +942us[ +944us] +/- 53ms 2024-04-25T15:24:04.501 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -1257us[-1257us] +/- 74ms 2024-04-25T15:24:04.501 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -112us[ -112us] +/- 67ms 2024-04-25T15:24:04.501 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:24:04.503 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:24:04.503 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-25T15:24:04.503 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +974us[ +994us] +/- 53ms 2024-04-25T15:24:04.503 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -1241us[-1241us] +/- 74ms 2024-04-25T15:24:04.503 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -90us[ -70us] +/- 68ms 2024-04-25T15:24:04.503 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:24:04.504 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T15:24:04.513 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T15:24:04.514 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T15:24:04.527 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T15:24:04.620 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T15:24:04.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:24:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:24:04.754 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:24:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:24:04.783 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T15:24:04.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T15:24:04.822 INFO:teuthology.task.internal:Duration was 630.813255 seconds 2024-04-25T15:24:04.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T15:24:04.873 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T15:24:04.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T15:24:04.876 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T15:24:04.911 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:24:04.915 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:24:05.208 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T15:24:05.209 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-04-25T15:24:05.209 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T15:24:05.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-25T15:24:05.230 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T15:24:05.269 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T15:24:05.270 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T15:24:05.272 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T15:24:05.336 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T15:24:05.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T15:24:05.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T15:24:05.477 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T15:24:05.488 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T15:24:05.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T15:24:05.511 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T15:24:05.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T15:24:05.578 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:24:05.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:24:05.600 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-04-25T15:24:05.602 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-25T15:24:05.614 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:24:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:24:05.667 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:24:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:24:05.681 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T15:24:05.692 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T15:24:05.693 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/remote/smithi146 2024-04-25T15:24:05.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T15:24:05.740 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201/remote/smithi177 2024-04-25T15:24:05.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T15:24:05.773 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T15:24:05.773 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T15:24:05.777 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T15:24:05.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T15:24:05.836 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T15:24:05.836 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T15:24:05.849 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T15:24:05.849 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T15:24:05.852 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T15:24:05.863 INFO:teuthology.orchestra.run.smithi146.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 15:24 /home/ubuntu/cephtest 2024-04-25T15:24:05.877 INFO:teuthology.orchestra.run.smithi177.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 15:24 /home/ubuntu/cephtest 2024-04-25T15:24:05.878 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T15:24:05.889 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T15:24:05.917 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T15:24:05.942 DEBUG:teuthology.nuke:shortname: smithi146 2024-04-25T15:24:05.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T15:24:05.966 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:07:57.871572', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQcOlVxd1/CY5+GNNxG/GW2bWEGfxv1T+F972WipdFff3Uoq5q2kf5snnF+Wr0YvTlLWQb0yOwELfzjQyBMzQ8='} 2024-04-25T15:24:05.969 DEBUG:teuthology.nuke:shortname: smithi177 2024-04-25T15:24:05.969 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T15:24:05.995 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:07:57.873359', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3cCg9gTCONg9DJ8ATtl2/Liuyg4DoCd5g0p6RrKYZZBcBCL5Xf3J/DMVLliAWXV3y76EzJas7ENCk58VwXd18='} 2024-04-25T15:24:06.020 INFO:teuthology.orchestra.console.smithi146:Power off 2024-04-25T15:24:06.021 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T15:24:06.046 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-04-25T15:24:06.046 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:06.066 INFO:teuthology.orchestra.console.smithi177:Power off 2024-04-25T15:24:06.066 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T15:24:06.075 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-04-25T15:24:06.088 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-04-25T15:24:06.088 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:06.111 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-25T15:24:10.077 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:10.112 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:10.191 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-04-25T15:24:10.223 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-25T15:24:14.194 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:14.225 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:14.305 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-04-25T15:24:14.335 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-25T15:24:14.406 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-04-25T15:24:18.338 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T15:24:18.448 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-04-25T15:24:18.549 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-04-25T15:24:18.654 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} duration: 630.8132548332214 failure_reason: 'Test failure: test_quiesce_dir_fragment (tasks.cephfs.test_quiesce.TestQuiesce)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=af2f4d86f1cb44569d03d0535c036ed9 status: fail success: false 2024-04-25T15:24:18.654 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:24:18.737 INFO:teuthology.run:FAIL